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I.  ABSTRACT  AND  SUMMARY 


An  “elevator  speech”  for  the  capabilities  delivered  by  the  Department  of  Defense  (DoD)  Systems 
Engineering  Research  Center  (SERC)  Systems  Engineering  (SE)  Effectiveness  Measurement  (EM)  task 
reads  as  follows: 

For  the  DoD,  whose  Major  Defense  Acquisition  Programs  (MDAPs)  frequently  and  significantly 
overrun  their  budgets  and  schedules  and  deliver  incomplete  systems,  the  SERC  SE  EM  framework, 
operational  concepts,  and  tools  will  empower  MDAP  sponsors  and  performers  to  collaboratively 
determine  their  early  SE  shortfalls  and  enable  the  development  of  successful  systems  within  their 
resource  constraints. 

Unlike  traditional  schedule-based  and  event-based  reviews,  the  SERC  SE  EM  technology  enables 
sponsors  and  perfonners  to  agree  on  the  nature  and  use  of  more  effective  evidence-based  reviews. 
These  enable  early  detection  of  missing  SE  capabilities  or  personnel  competencies  with  respect  to  a 
framework  of  Goals,  Critical  Success  Factors  (CSFs),  and  Questions  determined  by  the  EM  task  from 
the  leading  DoD  early-SE  CSF  analyses.  The  EM  tools  enable  risk-based  prioritization  of  corrective 
actions,  as  shortfalls  in  evidence  for  each  question  are  early  uncertainties,  which  when  combined  with 
the  relative  system  impact  of  a  negative  answer  to  the  question,  translates  into  the  degree  of  risk  that 
needs  to  be  managed  to  avoid  system  overruns  and  incomplete  deliveries. 


II.  INTRODUCTION 

The  mission  of  the  DoD  Systems  Engineering  Research  Center  (SERC)  is  to  perfonn  research  leading  to 
transfonnational  SE  methods,  processes,  and  tools  (MPTs)  that  enable  DoD  and  Intelligence 
Community  (IC)  systems  to  achieve  significantly  improved  mission  successes.  In  working  with  the 
DoD  EM  task  sponsors,  the  EM  team  converged  on  a  scope  and  direction  of  the  research  that  has 
created  and  piloted  a  set  of  EM  MPTs  that  has  strong  prospects  for  transforming  the  SE  process  into  an 
evidence-based  approach  that  enables  early  identification  and  resolution  of  program  risks. 

The  major  scope  decisions  worked  out  with  the  sponsors  and  their  rationales  are  summarized  in  Table  1. 
For  example,  the  original  Statement  of  Work  specified  coverage  of  multiple  system  types:  weapons 
platforms,  systems  of  systems,  and  net-centric  services;  but  converged  on  MDAPs,  as  these  were  the 
DoD  programs  most  likely  to  benefit  from  improved  SE  EMs.  Similarly,  it  focused  on  SE  EMs 
common  to  ground,  sea,  air,  and  space  systems  rather  than  trying  to  cover  them  all.  However,  the  EMs 
and  tools  are  designed  to  be  easy  to  extend  to  special  domains  by  special-domain  organizations.  The 
development  of  prototype  tools  was  performed  both  to  facilitate  pilot  testing  of  the  EM  framework  and 
questions,  but  also  to  provide  an  early  demonstration  that  the  SERC  research  could  lead  to  the 
development  of  useful  capabilities.  Initially  some  sponsors  were  interested  in  research  on  the  return  on 
investment  for  specific  SE  practices,  but  we  found  insufficient  data  to  support  such  analyses,  and  instead 
have  structured  the  tools  so  that  they  could  serve  as  ways  to  generate  such  data  for  the  future.  More 
detail  on  the  evolution  of  the  EM  project’s  plans  and  schedules  is  provided  in  Appendix  Z. 
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Table  1.  Summary  of  Major  Scope  Decisions 


Decision 

Rationale 

MDAP  vs.  multi-type  EMs 

SE  shortfalls  a  major  MDAP  problem 

Core  vs.  all-domain  EMs 

Avoid  numerous  inapplicable  EMs 

Ease  of  tailoring,  extension 

Enable  special-community  tailoring 

Cover  SE  functional  performance 

Sponsor  priority 

and  personnel  competency 

Rate  both  degree  of  impact  and 

Relation  to  risk  exposure  RE=P(L)*S(L),  ease 

degree  of  satisfaction  evidence 

of  tailoring  out  zero-impact  questions 

Hierarchical  goal  -  critical  success 

Ease  of  use,  understanding;  compatibility 

factor  -  question  framework 

with  related  frameworks 

Compatibility  with  INCOSE 

Complementary  coverage:  continuous  vs. 

Leading  Indicators 

discrete;  quantitative  vs.  qualitative 

Framework  and  tools 

Early  SERC  tangible  product 

Pilot  use  and  evaluation 

Evidence  of  strengths  and  shortfalls 

Initial  focus  on  project 

ROI  data  unavailable;  could  be  generated 

assessment  vs.  practice  ROIs 

via  tool  use 

Initial  focus  on  early  SE 

Highest  leverage  on  outcomes 

Section  III  summarizes  the  overall  methods,  assumptions,  and  procedures  used  in  the  EM  task.  Section 
IV  begins  with  an  analysis  of  the  DoD  needs,  opportunities,  and  business  case  for  the  use  of  SE  EM 
methods,  process,  and  tools  (MPTs).  It  finds  that  the  business  case  is  very  strong  for  large  MDAPs,  and 
not  very  strong  for  ad-hoc,  quick-response  system  development.  It  then  elaborates  the  detail  of  the  SE 
EM  framework  and  tools,  the  concepts  of  operation  for  their  use,  their  use  on  pilot  projects,  and  their 
analysis  in  comparison  with  the  Defense  Acquisition  Program  Support  (DAPS)  Methodology  and  its 
Systemic  Analysis  Database  (SADB)  of  DAPS  assessment  results.  It  concludes  that  the  evidence  of 
utility  and  business-case  return  on  investment  is  sufficient  to  proceed  toward  enabling  their  broad  DoD 
use,  subject  to  perfonning  further  research  on  the  factors  necessary  to  ensure  the  EM  MPTs’  scalability, 
extendability,  and  adaptability  to  change,  and  to  performing  the  resulting  improvements  to  the  EM 
MPTs.  It  recommends  a  two-phase  approach  for  achieving  an  initial  operational  capability  and 
transitioning  it  to  a  sustaining  organization.  This  would  begin  with  research  on  approaches  to  the  key 
issues,  and  continue  with  incremental  elaboration,  experimentation,  and  refinement  of  the  preferred 
approaches. 


III.  METHODS,  ASSUMPTIONS,  AND  PROCEDURES 
A.  Motivation  and  Context 

Numerous  General  Accountability  Office  reports,  Defense  Science  Board  and  National  Research 
Council  studies,  and  other  studies  have  addressed  the  magnitude  and  frequency  of  DoD  MDAP  budget 
and  schedule  overruns  and  delivery  deficiencies.  Many  of  these  have  identified  inadequate  SE  as  a 
major  source  of  the  problems.  Further  analyses  have  found  that  these  SE  inadequacies  have  largely 
resulted  from  commitments  to  proceed  based  on  inadequate  evidence  that  the  proposed  system  solutions 
can  actually  meet  DoD  needs  within  DoD’s  operational  environment  and  within  the  program’s  available 
budget  and  schedule  resources. 
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This  research  project  was  commissioned  by  the  SERC  sponsors  to  identify  and  organize  a  framework  of 
SE  effectiveness  measures  (EMs)  that  could  be  used  to  assess  the  evidence  that  a  MDAP’s  SE  approach, 
current  results,  and  personnel  competencies  were  sufficiently  strong  to  enable  program  success. 
Another  component  of  the  research  was  to  fonnulate  operational  concepts  that  would  enable  MDAP 
sponsors  and  performers  to  use  the  EMs  as  the  basis  of  collaborative  formulation,  scoping,  planning,  and 
monitoring  of  the  program’s  SE  activities,  and  to  use  the  monitoring  results  to  steer  the  program  toward 
the  achievement  of  feasible  SE  solutions. 


B.  Technical  Approach 

The  EM  research  project  reviewed  over  two-dozen  sources  of  candidate  SE  EMs,  and  converged  on  the 
strongest  sources  to  be  used  to  identify  candidate  SE  EMs.  We  developed  a  coverage  matrix  to 
determine  the  envelope  of  candidate  EMs,  and  the  strength  of  consensus  on  each  candidate  EM.  It  fed 
the  results  back  to  the  source  originators  to  validate  the  coverage  matrix  results.  This  resulted  in  further 
insights  and  added  candidate  EMs  to  be  incorporated  into  an  SE  Performance  Assessment  Framework. 
The  resulting  framework  is  organized  into  a  hierarchy  with  4  Goals,  18  Critical  Success  Factors,  and  74 
Questions  that  appeared  to  cover  the  central  core  of  common  SE  performance  detenninants  of  SE 
effectiveness. 

Concurrently,  the  research  project  was  extended  to  also  assess  SE  personnel  competency  as  a 
determinant  of  program  success.  We  analyzed  an  additional  six  personnel  competency  assessment 
frameworks  and  sets  of  questions.  Their  Goals  and  Critical  Success  Factors  were  very  similar  to  those 
used  in  the  SE  Performance  Assessment  Framework,  although  the  Questions  were  different.  The 
resulting  SE  Competency  Assessment  Framework  added  one  further  Goal  of  Professional  and 
Interpersonal  Skills  with  five  Critical  Success  Factors,  resulting  in  a  framework  of  5  Goals,  23  Critical 
Success  Factors,  and  81  Questions. 

In  order  to  evaluate  the  SE  Performance  and  Competency  Assessment  Frameworks,  we  developed 
simple,  easy-to-use  spreadsheet  tools  for  pilot  projects  to  use  and  provide  feedback  on  the  utility  of  the 
frameworks  and  tools.  The  tools  are  called  the  SE  Performance  Assessment  Tool  (SEP AT)  and  the  SE 
Competency  Assessment  Tool  (SEC AT).  The  initial  round  of  7  pilot  projects  yielded  quite  positive 
overall  assessment  result,  as  discussed  in  Sections  IV. B. 5  and  IV.D.  It  also  provided  several  valuable 
suggestions  for  improvement,  some  of  which  have  already  been  implemented. 

We  performed  a  complementary  analysis  comparing  the  coverage  of  the  SE  EMs  with  the  content  of  the 
DAPS  Methodology,  and  with  respect  to  the  initial  results  of  queries  to  the  Systemic  Analysis  Database 
(SADB)  of  negative  findings  resulting  from  DAPS  assessments.  The  results  again  were  largely  positive, 
as  discussed  in  Section  IV.E.  Overall,  the  DAPS  Methodology  goes  into  greater  detail  in  its  questions, 
providing  a  complementary  capability  for  users  of  the  SEPAT  and  SECAT  to  apply  in  focusing  in  on 
their  high-impact  questions. 

We  developed  operational  concepts  for  the  use  of  the  SEPAT  and  SECAT  on  MDAPs  for  three  early  life 
cycle  stages.  Each  involves  collaborative  efforts  by  the  program  sponsors  and  performers.  For  each 
question,  the  sponsors  rate  the  relative  impact  on  the  program  if  the  answer  is  negative,  and  a  scale  of 
Critical  (40-100%  overrun  or  its  equivalent  in  delivery  shortfalls);  Significant  (20-40%  overrun); 
Moderate  (2-20%  overrun);  and  Little  or  No  Impact  (0-2%  overrun).  These  ratings  are  provided  to  the 
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performers,  who  identify  situations  in  which  the  ratings  appear  inconsistent  with  the  program’s 
objectives,  available  resources,  contract  provisions,  or  likely  system  impact.  These  situations  stimulate 
constructive  discussion  and  a  more  compatible  shared  vision  and  feasible  set  of  impact  ratings  and 
program  parameters  to  be  used  in  managing  the  program. 

Once  consensus  is  reached  on  the  impact  ratings,  the  performer  SEs  proceed  to  define  and  develop  the 
system,  along  with  evidence  that  the  questions  are  being  satisfactorily  addressed.  The  evidence  is 
evaluated  by  independent  experts  at  each  major  review.  Shortfalls  in  evidence  are  identified  as 
uncertainties  and  risk  probabilities,  which  when  combined  with  the  question’s  risk  impact  detennines 
the  level  of  risk  exposure  associated  with  the  question.  To  reinforce  the  importance  of  the  evidence- 
based  assessments,  a  considerable  portion  of  the  perfonner’s  award  fee  is  based  on  the  degree  to  which 
the  performer  evidence  has  shown  the  project  to  be  at  a  low  level  of  risk. 

The  evidence  is  thus  a  first-class  deliverable,  as  compared  to  being  an  optional  appendix  in  most  current 
programs,  whose  content  is  largely  dropped  as  project  resources  become  scarce.  As  such,  its 
development  needs  to  be  planned,  budgeted,  and  made  a  key  element  in  the  project’s  earned  value 
management  system.  Any  risks  resulting  from  shortfalls  in  evidence  need  to  be  addressed  by  risk 
mitigation  plans  and  their  associated  resource  requirements.  Both  evidence  generation  and  risk 
mitigation  add  up-front  effort,  but  a  business  case  analysis  of  the  effects  of  going  from  minimal  to 
thorough  architecture  and  risk  resolution  on  161  software-intensive  systems  yields  significant  returns  on 
investment  for  MDAP-scale  projects,  as  discussed  in  Section  IV. A. 3. 


IV.  RESULTS  AND  DISCUSSION 

This  Section  begins  with  a  summary  of  the  DoD  needs  for  improvements  in  overall  systems  acquisition, 
and  particularly  in  systems  engineering.  One  of  the  particular  needs  in  DoD  systems  engineering  is  for 
better  ways  for  measuring  its  effectiveness,  as  it  is  difficult  to  manage  something  that  one  is  not  able  to 
measure.  A  particular  opportunity  is  for  SE  EMs  that  focus  on  measurable  and  independently  verifiable 
evidence  of  effectiveness,  and  associated  evidence-based  reviews  that  enable  MDAPs  to  identify 
shortfalls  in  evidence  of  the  feasibility  of  the  specifications  and  plans  being  presented  for  approval  at 
milestone  reviews.  This  opportunity  led  us  to  associate  the  SE  EMs  with  operational  concepts 
highlighting  their  use  in  evidence-based  reviews.  Section  A  also  presents  a  business  case  for  the 
thoroughness  of  SE  activities,  which  shows  that  the  payoff  for  evidence-based  SE  EMs  is  greatest  for 
large  MDAPs. 

Section  B  proceeds  to  describe  the  derivation  of  the  SE  EM  frameworks  of  Goals,  Critical  Success 
Factors,  and  Questions  for  performance  and  personnel  competency  from  leading  DoD  studies.  It  then 
describes  the  Systems  Engineering  Performance  Assessment  Tool  (SEP AT)  and  Systems  Engineering 
Competency  Assessment  Tool  (SECAT)  tools  developed  to  support  pilot  evaluation  of  the  SE  EM 
frameworks,  and  then  summarizes  the  results  of  the  pilot  studies.  Section  C  then  presents  several 
operation  all  concepts  for  using  the  tools,  both  at  major  DoD  milestones  and  during  planning  and 
execution  of  DoD  MDAP  projects.  Section  D  provides  a  more  detailed  summary  of  two  of  the  pilot 
evaluations,  and  Section  E  summarizes  a  complementary  evaluation  of  the  framework  via  its 
comparison  to  the  DoD  Defense  Acquisition  Program  Support  (DAPS)  methodology  and  its  associated 
Systemic  Analysis  Database  (SADB)  of  DAPS  program  assessment  results. 
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Section  F  summarizes  the  conclusions  that  the  business  case,  the  pilot  results,  and  the  DAPS  and  SADB 
comparison  results  all  point  to  a  strong  payoff  for  DoD  use  of  the  SE  SM  framework,  tools,  and 
evidence-based  concepts  of  operation.  It  also  provides  recommendations  for  a  two-phase  approach  for 
achieving  an  initial  operational  capability  and  transitioning  it  to  a  sustaining  organization.  This  would 
begin  with  research  on  approaches  to  the  key  issues,  and  continue  with  incremental  elaboration, 
experimentation,  and  refinement  of  the  preferred  approaches. 

A.  DoD  MDAP  SE  EM  Needs,  Opportunities,  and  Business  Case 

1.  The  Need  and  Opportunity 

Table  2,  obtained  from  a  recent  General  Accountability  Office  (GAO)  report  [29],  shows  the  magnitude 
of  the  problems  to  be  addressed  by  the  SE  EM  capabilities.  These  included  total  DoD  annual  MDAP 
cost  growths  of  roughly  $300  billion  per  year  and  delivery  delays  coming  close  to  two  years.  In  many 
cases,  these  “cost  and  schedule  growths”  were  not  actual  growths,  but  the  results  of  traditional 
acquisition  practices  requiring  programs  to  associate  costs  and  schedules  with  capabilities  before 
evidence  of  technical,  cost,  and  schedule  feasibility  was  available. 


Table  2.  Analysis  of  U.S.  Defense  Dept.  Major  Defense  Acquisition  Program  Portfolios 


Analysis  of  U.S.  Defense  Department  Major  Defense  Acquisition  Program  Portfolios 

Fiscal  2009  dollars 

Portfolio  size 

2003 

2007 

2008 

Number  of  programs 

77 

95 

96 

Total  planned  commitments 

$1.2  trillion 

$1.6  trillion 

$1.6  trillion 

Commitments  outstanding 

$724.2  billion 

$875.2  billion 

$786.3  billion 

Portfolio  indicators 

Change  to  total  RDT&E*  costs  from  first 
estimate 

37% 

40% 

42% 

Change  to  total  acquisition  costs  from  first 
estimate 

19% 

26% 

25% 

Total  acquisition  cost  growth 

$183  billion 

$301.3  billion 

$296.4  billion 

Share  of  programs  with  25%  increase 
in  program  acquisition  unit  cost  growth 

41% 

44% 

42% 

Average  schedule  delay  in  delivering  initial 
capabilities 

18  months 

21  months 

22  months 

Source:  U.S.  Government  Accountability  Office  ^Research,  Development,  Testing  &  Evaluation 

Providing  such  validated  evidence  is  generally  considered  to  be  a  good  practice,  but  generally  fails  to  be 
done  well.  This  is  because  of  a  lack  of  evidence  criteria;  a  lack  of  evidence-generation  procedures  and 
measures  for  monitoring  evidence  generation;  a  lack  of  appreciation  of  the  consequences  of  proceeding 
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into  development  with  unsubstantiated  specifications  and  plans;  and  because  of  current  methods, 
standards,  and  contractual  provisions  that  make  evidence  generation  optional.  The  main  contributions 
of  the  SERC  SE  EM  task  are  to  provide  experience-based  approaches  for  each  of  these  concerns  and  to 
illustrate  the  consequences  of  their  use  via  case  studies  and  parametric  analysis. 

2.  Evidence  Shortfalls  in  Current  SE  Practices 

2.1  Technical  Shortfalls 

Current  software  design  and  development  methods  focus  strongly  on  the  inputs  and  outputs, 
preconditions  and  post-conditions  that  a  system  function,  component,  or  service  operates  by  as  a 
product.  They  lack  adequate  capabilities  to  support  evidence  about  how  well  the  elements  perfonn,  how 
expensive  they  will  be  to  develop,  or  how  compatible  are  their  underlying  assumptions.  In  principle, 
they  support  reasoning  about  off-nominal  performance,  but  in  practice  their  descriptions  generally  focus 
on  sunny-day  scenarios.  As  a  result,  many  DoD  MDAP  project  reviews  tend  to  focus  on  exhaustive 
presentations  of  PowerPoint  charts  and  Unified  Modeling  Language  (UML)  diagrams.  They  provide 
little  evidence  that  the  system  they  describe  could  handle  rainy-day  scenarios;  perform  adequately  on 
throughput,  response  time,  safety,  security,  usability,  or  other  desired  quality  attributes  across  a  range  of 
representative  mission  scenarios;  be  buildable  within  the  available  budgets  and  schedules  in  the  plan;  or 
generate  positive  returns  on  investment  for  the  stakeholders. 

Figure  1  shows  a  summary  from  a  2007  National  Defense  Industrial  Association  (NDIA)  workshop  of 
analyzing  the  underlying  causes  of  the  negative  findings  of  DAPS  reviews  of  several  dozen  DoD 
programs.  It  shows  that  two  of  the  top  four  causes  of  negative  findings  were  due  to  SE  shortfalls  in 
technical  processes  and  requirements  processes,  shown  in  bold.  Several  others,  shown  in  italics,  were 
partly  due  to  inadequate  SE  perfonnance,  often  caused  by  the  fact  that  adequate  SE  is  the  first  victim  of 
a  lack  of  program  realism  with  respect  to  the  necessary  budgets  and  schedules  needed  to  do  a  thorough 
job. 


SE  Performance,  Competency  are  Major  Sources  of 

OSD/AT&L  Systemic  Analysis  Negative  Findings 

1  Technical  process  (35  instances) 

6  Lack  of  appropriate  staff  (23) 

-  V&V,  integration,  modeling  &  sim. 

2  Management  process  (31 ) 

7  Ineffective  organization  (22) 

3  Acquisition  practices  (26) 

8  Ineffective  communication  (21) 

4  Requirements  process  (25) 

9  Program  realism  (21) 

5  Competing  priorities  (23) 

10  Contract  structure  (20) 

Figure  1.  Analysis  of  Negative  Findings  of  DAPS  Reviews 
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2.2  Management  Shortfalls 


A  major  recent  step  forward  in  the  management  of  outsourced  government  projects  has  been  to  move 
from  schedule-based  reviews  to  event-based  reviews.  A  schedule-based  review  says  basically  that:  “The 
contract  specifies  that  the  Preliminary  Design  Review  (PDR)  will  be  held  on  June  1,  2011,  whether  we 
have  a  design  or  not.” 

In  general,  neither  the  customer  nor  the  developer  wants  to  fail  the  PDR,  so  the  project  goes  into 
development  with  the  blessing  of  having  passed  a  PDR,  but  with  numerous  undefined  interfaces  and 
unresolved  risks.  As  shown  below,  these  will  generally  result  in  major  project  overruns  and  incomplete 
deliveries. 

An  event-based  review  says:  “Once  we  have  a  preliminary  design,  we  will  hold  the  PDR.”  Such  a 
review  will  generally  consist  of  exhaustive  presentations  of  sunny-day  PowerPoint  charts  and  UML 
diagrams.  This  is  largely  because  most  traditional  DoD  acquisition  contracts  and  procedures  have  an 
early  focus  on  product-oriented  versus  feasibility-evidence-oriented  deliverables  and  reviews. 

These  reinforce  paths  toward  project  disaster,  as  in  this  quote  from  a  recent  large-project  manager:  “I’d 
like  to  have  some  of  my  systems  engineers  address  those  software  quality-factor  risks,  but  my  contract 
deliverables  and  award  fees  are  based  on  having  all  of  the  system’s  functions  defined  by  the  next 
review.”  Similar  over-focus  on  product  definition  is  found  in  project  earned-value  management  systems 
for  tracking  project  progress  and  Data  Item  Descriptions  (DIDs)  for  deliverables.  Most  contract  DIDs 
cover  function,  interface,  and  infrastructure  considerations,  but  place  demonstration  of  their  feasibility 
in  optional  appendices  where,  as  with  the  project  manager  above,  they  are  the  first  to  go  when  time  and 
effort  are  running  out. 


3.  Consequences  of  Evidence  Shortfalls 

This  is  not  just  a  DoD  problem,  but  a  pervasive  problem  in  system  acquisition  and  outsourcing.  For 
example,  the  biannual  Standish  Reports  consistently  identify  shortfalls  in  evidence  of  feasibility  with 
respect  to  stakeholder  objectives  as  the  major  root  causes  of  project  failure.  The  2009  Standish  Report 
[29]  found  that  only  32%  of  the  9000  projects  reported  delivered  their  full  capability  within  their  budget 
and  schedule;  24%  were  cancelled;  and  44%  were  significantly  over  budget,  over  schedule,  and/or 
incompletely  delivered.  More  detail  on  the  top  critical  success  factors  distinguishing  successful  from 
failed  software  projects  was  in  the  2005  Standish  Report.  There,  71%  of  the  sources  of  failure  were 
primarily  due  to  evidence  shortfalls  with  respect  to  stakeholder  objectives  (lack  of  user  involvement, 
executive  support,  clear  requirements,  proper  planning,  and  realistic  expectations). 

Recent  further  analyses  of  the  Constructive  Cost  Model  (COCOMO)  II  database  on  the  effects  of 
incomplete  architecture  definition,  risk  resolution,  and  resulting  feasibility  evidence  on  software¬ 
intensive  systems  are  shown  in  Figure  2.  They  show  the  results  of  a  risk-driven  “how  much  architecting 
is  enough?”  analysis,  based  on  the  COCOMO  II  Architecture  and  Risk  Resolution  (RESL)  factor  [8], 
[13].  This  factor  was  calibrated  along  with  22  others  to  161  project  data  points.  It  relates  the  amount  of 
extra  rework  effort  on  a  project  to  the  degree  of  evidence  that  the  project  had  demonstrated  its 
architecture  feasibility  and  resolved  its  technical  and  management  risks.  This  also  correlates  with  the 
percent  of  project  effort  devoted  to  software-intensive  system  architecting  and  risk  resolution. 
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Effects  of  Size  on  Sweet  Spots  Effects  of  Volatility  and  Criticality 


Figure  2.  Architecture  Evidence  Shortfall  Penalties  and  Resulting  Architecting  Investment  Sweet 

Spots 

The  analysis  indicated  that  the  amount  of  rework  was  an  exponential  function  of  project  size.  A  small 
(10  thousand  equivalent  source  lines  of  code,  or  KSLOC)  project  could  fairly  easily  adapt  its 
architecture  to  rapid  change  via  refactoring  or  its  equivalent,  with  a  rework  penalty  of  18%  between 
minimal  and  extremely  thorough  architecture  and  risk  resolution.  However,  a  very  large  (10,000 
KSLOC)  project  would  incur  a  corresponding  rework  penalty  of  91%,  covering  such  effort  sources  as 
integration  rework  due  to  undiscovered  large-component  interface  incompatibilities,  technology 
immaturities,  and  critical  performance  shortfalls. 

The  effects  of  rapid  change  (volatility)  and  high  dependability  (criticality)  on  the  architecture  evidence 
shortfall  penalties  and  resulting  architecting  investment  sweet  spots  are  shown  in  the  right  hand  graph. 
Here,  the  solid  black  lines  represent  the  average-case  cost  of  rework,  architecting,  and  total  cost  for  a 
100-KSLOC  project  as  shown  at  the  left.  The  dotted  red  lines  show  the  effect  on  the  cost  of 
architecting  and  total  cost  if  rapid  change  adds  50%  to  the  cost  of  architecture  and  risk  resolution. 
Quantitatively,  this  moves  the  sweet  spot  from  roughly  20%  to  10%  of  effective  architecture  investment 
(but  actually  15%  due  to  the  50%  cost  penalty).  Thus,  high  investments  in  architecture,  feasibility 
analysis,  and  other  documentation  do  not  have  a  positive  return  on  investment  for  very  high-volatility 
projects  due  to  the  high  costs  of  documentation  rework  for  rapid-change  adaptation. 

The  dashed  blue  lines  at  the  right  represent  a  conservative  analysis  of  the  cost  effects  of  system  failure 
due  to  unidentified  architecting  shortfalls.  It  assumes  that  the  costs  of  architecting  shortfalls  are  not 
only  added  rework,  but  also  losses  to  the  organization’s  operational  effectiveness  and  productivity. 
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These  are  conservatively  assumed  to  add  50%  to  the  project-rework  cost  of  architecture  shortfalls  to  the 
organization.  In  most  cases  for  high-assurance  systems,  the  added  cost  would  be  considerably  higher. 

Quantitatively,  this  moves  the  sweet  spot  from  roughly  20%  to  over  30%  as  the  most  cost-effective 
investment  in  architecting  and  development  of  feasibility  evidence  for  a  100-KSLOC  project.  It  is  good 
to  note  that  the  “sweet  spots”  are  actually  relatively  flat  “sweet  regions”  extending  5-10%  to  the  left  and 
right  of  the  “sweet  spots.”  However,  moving  to  the  edges  of  a  sweet  region  increases  the  risk  of 
significant  losses  if  some  project  assumptions  turn  out  to  be  optimistic. 

The  bottom  line  for  Figure  2  is  that  the  greater  the  project’s  size,  criticality,  and  stability  are,  the  greater 
is  the  need  for  validated  architecture  feasibility  evidence.  However,  for  very  small,  low-criticality 
projects  with  high  volatility,  the  evidence-producing  efforts  would  make  little  difference  and  would 
need  to  be  continuously  redone,  producing  a  negative  return  on  investment.  In  such  cases,  agile 
methods  such  as  rapid  prototyping,  Scrum  [27]  and  extreme  Programming  [3]  will  be  more  effective. 
Overall,  evidence-based  specifications  and  plans  are  a  much  better  match  to  MDAPs,  where  in  general 
they  will  eliminate  many  of  the  system  delivery  overruns  and  shortfalls  experienced  on  current  MDAPs. 


B.  DoD  MDAP  SE  EM  Framework  and  Tools:  SEPAT  and  SECAT 

Our  research  suggests  that  systems  engineering  (SE)  effectiveness  measures  (EMs)  can  be  characterized 
along  two  major  dimensions  of  assessment,  and  across  two  timescales.  SE  effectiveness  can  be  assessed 
both  by  the  performance  of  the  SE  function,  and  by  the  competency  of  those  perfonning  it.  Each 
dimension  can  be  evaluated  at  discrete  decision  points  in  a  program,  and  also  in  a  continuous  fashion 
throughout  its  execution. 

We  propose  three  frameworks  to  evaluate  SE  EMs  along  these  dimensions  and  timescales,  and  have 
created  prototype  tools  to  evaluate  two  of  these  frameworks  in  the  discrete  dimension:  the  Systems 
Engineering  Performance  Assessment  Tool  (SEPAT),  and  the  Systems  Engineering  Competency 
Assessment  Tool  (SECAT).  Complementing  these  discrete-evaluation  frameworks,  research  by 
INCOSE  on  continuous  evaluation  has  led  to  conceptual  development  of  a  set  of  leading  indicators  (LI), 
which  were  included  as  candidates  in  developing  the  SEPAT  performance  evaluation  questions  (See 
Table  3.) 


Table  3.  Dimensions  and  timescales  of  EM  evaluation 


Discrete 

SEPAT 

SECAT 

Continuou 

s 

INCOSE  Leading  Indicators 

Performance 

Competency 

This  section  discusses  the  evolution  of  the  SE  EM  evaluation  frameworks,  and  the  specific 
implementations  of  these  frameworks  into  the  prototype  SEPAT  and  SECAT  tools.  We  also  present 
preliminary  results  of  pilot  evaluations  of  the  SEPAT  and  SECAT  tools,  which  were  developed  to  help 
determine  the  utility  of  the  proposed  EM  frameworks  across  multiple  acquisition  frameworks  and 
application  domains. 


9 


1.  SEPAT  and  SECAT  Goal-Critical  Success  Factor-Question  Framework 


Our  initial  research  focused  on  identifying  methods  that  might  be  suitable  for  assessing  the  effectiveness 
of  systems  engineering  on  major  defense  acquisition  programs  (MDAPs).  A  literature  review  identified 
eight  candidate  measurement  methods,  as  follows: 

•  NRC  Pre -Milestone  A  &  Early-Phase  SysE  top-20  checklist  [20] 

•  Air  Force  Probability  of  Program  Success  (PoPS)  Framework  [1] 

•  INCOSE/LMCO/MIT  Leading  Indicators  [24] 

•  Stevens  Leading  Indicators  (new;  using  SADB  root  causes)  [34] 

•  USC  Anchor  Point  Feasibility  Evidence  progress  [31] 

•  UAH  teaming  theories  [14] 

•  NDIA/SEI  capability/challenge  criteria  [15] 

•  SISAIG  Early  Warning  Indicators  [9]  /  USC  Macro  Risk  Tool  [33] 

Pages  5-8  of  the  NRC  report  [20]  suggests  a  “Pre -Milestone  A/B  Checklist”  forjudging  the  successful 
completion  of  early-phase  systems  engineering.  Using  this  checklist  as  a  concise  starting  point,  the 
researchers  identified  similar  key  elements  in  each  of  the  other  candidate  measurement  methods, 
resulting  in  a  list  of  44  characteristics  of  effective  systems  engineering  (see  Appendix  C).  Multiple 
research  teams  independently  examined  two  or  more  candidate  EMs  to  assess  whether  and  to  what 
degree  each  characteristic  was  addressed  by  the  respective  measure,  as  noted  in  Table  4.  This 
assessment  also  identified  another  six  EM  characteristics  not  previously  noted. 


Table  4.  Review  of  candidate  EMs  by  research  teams 


Candidate  EM 

USC 

Stevens 

FC-MD 

UAH 

PoPS  Leading  Indicators  (Lis) 

X 

X 

X 

INCOSE  Us 

X 

X 

Stevens  Lis 

X 

X 

X 

SISAIG  Lis/  Macro  Risk 

X 

X 

X 

NRC  Top-20  List 

X 

X 

X 

SEI  CMMI-Based  Lis 

X 

X 

X 

USC  AP-Feasibility  Evidence 

X 

X 

X 

UAH  Team  Effectiveness 

X 

X 

X 
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Previous  research  by  the  USC  team  into  a  macro-risk  model  for  large-scale  projects  had  resulted  in  a 
taxonomy  of  high-level  goals  and  supporting  critical  success  factors  (CSFs)  based  on  [28].  This  was 
identified  as  a  potential  framework  for  organizing  the  51  EM  characteristics  identified  above.  Analysis 
of  the  characteristics  showed  that  they  could  be  similarly  organized  into  a  series  of  four  high-level  goals, 
each  containing  4-5  CSFs,  as  seen  in  Table  5.  Our  survey  of  the  existing  literature  suggests  that  these 
CSFs  are  among  the  factors  that  are  most  critical  to  successful  SE,  and  that  the  degree  to  which  the  SE 
function  in  a  program  satisfies  these  CSFs  is  a  measure  of  SE  effectiveness. 

Table  5.  Goals  and  CSFs  for  SE  performance 


High-level  Goals 

Critical  Success  Factors 

Concurrent 
definition  of 
system 

requirements  & 
solutions 

Understanding  of  stakeholder  needs 

Concurrent  exploration  of  solutions 

System  scoping  &  requirements 
definition 

Prioritization/allocation  of  requirements 

System  life-cycle 
organization, 
planning  &  staffing 

Establishment  of  stakeholder  RAAs 

Establishment  of  IPT  RAAs 

Establishment  of  resources  to  meet 
objectives 

Establishment  of 
selection/contracting/incentives 

Assurance  of  necessary  personnel 
competencies 

Technology 
maturing  & 
architecting 

COTS/NDI  evaluation,  selection, 
validation 

Life-cycle  architecture  definition  & 
validation 

Use  of  prototypes,  models,  etc.  to 
validate  maturity 

Validated  budgets  &  schedules 

Evidence-based 
progress 
monitoring  & 
commitment 
reviews 

Monitoring  of  system  definition 

Monitoring  of  feasibility  evidence 
development 

Monitoring/assessment/re-planning  for 
changes 

Identification  and  mitigation  for 
feasibility  risks 

Reviews  to  ensure  stakeholder 
commitment 

Related  to  the  effectiveness  measures  of  SE  performance  is  the  need  to  measure  the  effectiveness  of  the 
staff  assigned  to  the  SE  function.  Besides  the  eight  SEP  AT  sources,  six  additional  sources  were 
reviewed  for  contributions  to  Personnel  Competency  evidence  questions.  These  were: 

•  Office  of  the  Director  of  National  Intelligence  (ODNI),  Subdirectory  Data  Collection  Tool:  Systems 
Engineering  [22] 
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•  INCOSE  Systems  Engineering  Handbook,  August  2007  [17] 

•  ASN  (RD&A),  Guidebook  for  Acquisition  of  Naval  Software  Intensive  Systems,  September  2008 

[3] 

•  CMU/SEI,  Models  for  Evaluating  and  Improving  Architecture  Competence  [4] 

•  NASA  Office  of  the  Chief  Engineer,  NASA  Systems  Engineering  Behavior  Study,  October  2008 
[34] 

•  National  Research  Council,  Human-System  Integration  in  the  System  Development  Process,  2007 
[23] 

These  were  analyzed  for  candidate  knowledge,  skills,  and  abilities  (KSA)  attributes  proposed  for 
systems  engineers.  Organizing  these  work  activities  and  KSAs  revealed  that  the  first  four  goals  and 
their  CSFs  were  in  common  with  the  EM  taxonomy.  An  additional  goal  and  its  related  CSFs  were  also 
discovered,  as  presented  in  Table  6. 

Table  6.  Additional  goals  and  CSFs  for  SE  competency 


High-level  Goal 

Critical  Success  Factors 

Professional  and 
interpersonal  skills 

Ability  to  plan,  staff,  organize,  team- 
build,  control,  and  direct  systems 
engineering  teams 

Ability  to  work  with  others  to  negotiate, 
plan,  execute,  and  coordinate 
complementary  tasks  for  achieving 
program  objectives 

Ability  to  perform  timely,  coherent,  and 
concise  verbal  and  written 
communication 

Ability  to  deliver  on  promises  and 
behave  ethically 

Ability  to  cope  with  uncertainty  and 
unexpected  developments,  and  to  seek 
help  and  fill  relevant  knowledge  gaps 

2.  Question  Impact/Evidence  Ratings  and  Project  SE  Risk  Assessment 

Using  these  relatively  high-level  criteria,  however,  it  is  difficult  to  evaluate  whether  the  SE  on  a 
particular  program  adequately  satisfies  the  CSFs.  In  its  approach  to  evaluating  macro-risk  in  a  program, 
[31]  suggests  that  a  goal-question-metric  (GQM)  approach.  [4]  provides  a  method  to  accomplish  this 
evaluation.  Following  this  example,  the  researchers  developed  questions  to  explore  each  goal  and  CSF, 
and  devised  metrics  to  detennine  the  relevance  of  each  question  and  the  quality  of  each  answer. 

The  researchers  began  question  development  for  the  SE  perfonnance  framework  with  the  checklist  from 
[20].  Further  questions  were  adapted  from  the  remaining  EM  characteristics,  rewritten  as  necessary  to 
express  them  in  the  form  of  a  question.  Each  question  is  phrased  such  that,  answered  affirmatively,  it 
indicates  positive  support  of  the  corresponding  CSF.  We  hypothesize  that  the  strength  of  support  for 
each  answer  is  related  to  the  relative  risk  probability  associated  with  the  CSF  that  question  explores. 
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Rather  than  rely  simply  on  the  opinion  of  the  evaluator  as  to  the  strength  of  the  response,  a  more 
quantifiable  evidence-based  approach  was  selected.  The  strength  of  the  response  is  related  to  the 
amount  of  evidence  available  to  support  an  affirmative  answer — the  stronger  the  evidence,  the  lower  the 
risk  probability.  Feedback  from  industry,  government,  and  academic  participants  in  workshops 
conducted  in  March  and  May  2009  suggested  that  a  simple  risk  probability  scale  with  four  discrete 
values  be  employed  for  this  purpose. 

Evidence  takes  whatever  form  is  appropriate  for  the  particular  question.  For  example,  a  simulation 
model  might  provide  evidence  that  a  particular  performance  goal  can  be  met.  Further,  the  strongest 
evidence  is  that  which  independent  expert  evaluators  have  validated. 

Recognizing  that  each  characteristic  might  be  more  or  less  applicable  to  a  particular  program  being 
evaluated,  the  questions  are  also  weighted  according  to  the  risk  impact  that  failure  to  address  the 
question  might  be  expected  to  have  on  the  program.  Again  based  on  workshop  feedback,  a  four-value 
scale  for  impact  was  chosen. 

The  product  of  the  magnitude  of  a  potential  loss  (the  risk  impact)  and  the  likelihood  of  that  loss  (the  risk 
probability)  is  the  risk  exposure.  Although  risk  exposure  is  generally  calculated  given  quantitative  real- 
number  estimates  of  the  magnitude  and  probabilities  of  a  loss,  the  assessments  of  risk  impact  and  risk 
probability  described  above  use  an  ordinal  scale.  Therefore,  we  employ  a  mapping  between  the  four- 
value  risk  probability  and  risk  impact  scales  to  a  discrete  five-value  risk  exposure. 


3.  Prototype  SE  Effectiveness  Assessment  Tools 

As  a  means  to  test  the  utility  of  these  characteristics  for  assessing  systems  engineering  effectiveness, 
using  the  GQM  approach  outlined  above,  the  researchers  created  prototype  tools  that  might  be  used  to 
perform  periodic  evaluations  of  a  project,  similar  to  a  tool  used  in  conjunction  with  the  macro-risk 
model  described  above.  The  following  section  describes  this  prototype  implementation  in  further  detail. 


3.1  SE  Performance  Assessment  Tool 

The  systems  engineering  performance  assessment  tool  (SEP AT)  is  an  Excel  spreadsheet-based  prototype 
focused  on  enabling  projects  to  detennine  their  relative  risk  exposure  due  to  shortfalls  in  their  SE 
performance  relative  to  their  prioritized  project  needs.  It  complements  other  SE  perfonnance 
effectiveness  assessment  capabilities  such  as  the  INCOSE  Leading  Indicators,  in  that  it  supports 
periodic  assessment  of  evidence  of  key  SE  function  performance,  as  compared  to  supporting  continuous 
assessment  of  key  project  SE  quantities  such  as  requirements  volatility,  change  and  problem  closure 
times,  risk  handling,  and  staffing  trends. 

The  operational  concept  of  the  SEP  AT  tool  is  to  enable  project  management  (generally  the  Project 
Manager  or  his/her  designate)  to  prioritize  the  relative  impact  on  the  particular  project  of  shortfalls  in 
performing  the  SE  task  represented  in  each  question.  Correspondingly,  the  tool  enables  the  project 
systems  engineering  function  (generally  the  Chief  Engineer  or  Chief  Systems  Engineer  or  their 
designate)  to  evaluate  the  evidence  that  the  project  has  adequately  perfonned  that  task.  This 
combination  of  impact  and  risk  assessment  enables  the  tool  to  estimate  the  relative  project  risk  exposure 
for  each  question,  and  to  display  them  in  a  color-coded  Red- Yellow-Green  fonn. 
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These  ideas  were  reviewed  in  workshops  with  industry,  government,  and  academic  participants 
conducted  in  March  and  May  2009,  with  respect  to  usability  factors  in  a  real  project  environment.  A 
consensus  emerged  that  the  scale  of  risk  impact  and  risk  probability  estimates  should  be  kept  simple  and 
easy  to  understand.  Thus  a  red,  yellow,  green,  and  grey  scale  was  suggested  to  code  the  risk  impact;  and 
a  corresponding  red,  yellow,  green,  and  blue  scale  to  code  the  risk  probability.  These  scales  are 
discussed  in  more  depth  below.  An  example  of  the  rating  scales,  questions,  and  calculated  risk  exposure 
in  the  prototype  tool  is  presented  below. 


Impact 


Evidence/Risk 


S  8 


ts 

a 

E 


S  s 
S.  5? 


NOTE:  Impact  and  evidence/risk  ratings  should  be  done  independently.  The 
impact  rating  should  estimate  the  effect  a  failure  to  address  the  specified  item 
might  have  on  the  program.  The  evidence  rating  should  specify  the  qualtity  of 
evidence  that  has  been  provided,  which  demonstrates  that  the  specified  risk 
item  has  been  satisfactorily  addressed. 


Risk 

Exposure 


Goal  1: 


Concurrent  definition  of  system  requirements  and  solutions 


Understanding  of  stakeholder  needs:  capabilities,  operational  concept,  key 
performance  parameters,  enterprise  fit  (legacy) 

At  Milestone  A,  have  the  KPPs  been  identified  in  clear,  comprehensive,  concise  terms  that 
are  understandable  to  all  stakeholders? 

Has  a  CONOPS  been  developed  showing  that  the  system  can  be  operated  to  handle  both 
nominal  and  off-nominal  workloads,  to  meet  response  time  requirements,  and  generally 
to  meet  the  defined  KPPs? 

Has  the  ability  of  the  system  to  meet  mission  effectiveness  goals  been  verified  through 
the  use  of  modeling  and  simulation? 

Have  the  success-critical  stakeholders  been  identified,  their  roles  and  responsibilities 
negotiated,  and  their  needs  clearly  represented  by  the  KPPs  and  CONOPS? 

Have  issues  about  the  fit  of  the  system  into  the  stakeholders'  context  --  acquirers,  end 
users,  administrators,  interoperators,  maintainers,  etc.  --  been  adequately  explored? 


Figure  3.  Example  of  SEP  AT  prototype 


Risk  impact  ratings  vary  from  a  critical  impact  of  shortfalls  in  performing  the  SE  task  in  question  (red) 
through  significant  impact  (yellow)  and  moderate  impact  (green)  to  little-no  impact  (gray),  as  illustrated 
in  Table  7.  These  relative  impact  ratings  enable  projects  to  tailor  the  evaluation  to  the  project’s  specific 
situation.  Thus,  for  example,  it  is  easy  to  “drop”  a  question  by  clicking  on  its  “No  Impact”  button,  but 
also  easy  to  restore  it  by  clicking  on  a  higher  impact  button.  The  rating  scale  for  the  impact  level  is 
based  on  the  user’s  chosen  combination  of  effects  on  the  project’s  likely  cost  overrun,  schedule  overrun, 
and  missing  percent  of  promised  over  actual  delivered  capability  (considering  there  are  various  tradeoffs 
among  these  quantities). 


Table  7.  Risk  impact  ratings 


Rating 

Cost/Schedule/Capability  Shortfall 

Little-No  impact  (Gray) 

0-2%  (1%  average) 

Moderate  impact  (Green) 

2-20%(ll%  average) 

Significant  impact  (Yellow) 

20-40%  (30%  average) 

Critical  impact  (Red) 

40-100%  (70%  average) 
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Using  Question  1.1(a)  from  Figure  3  as  an  example,  if  the  project  were  a  back-room  application  for  base 
operations  with  no  mission-critical  key  performance  parameters  (KPPs),  its  impact  rating  would  be 
Little-No  impact  (Gray).  However,  if  the  project  were  a  C4ISR  system  with  several  mission-critical 
KPPs,  its  rating  would  be  Critical  impact  (Red). 

The  Evidence/Risk  rating  is  the  project’s  degree  of  evidence  that  each  SE  effectiveness  question  is 
satisfactorily  addressed,  scored  (generally  by  the  project  Chief  Engineer  or  Chief  Systems  Engineer  or 
their  designate)  on  a  risk  probability  scale:  the  less  evidence,  the  higher  the  probability  of  shortfalls.  The 
evaluator  chooses  a  rating  based  on  the  probability  of  an  unsuccessful  outcome  in  performing  the  SE 
task  in  question,  as  noted  in  Table  8 

Table  8.  Risk  probability/evidence  ratings 


Likelihood  of  Shortfall 

Degree  of  evidence 

Probability  Range 

High  probability  (Red) 

Little-no  evidence 

P  =  0.4  - 1.0;  average  0.7 

Medium  probability  (Yellow) 

Weak  evidence 

P  =  0.2-  0.4;  average  0.3 

Low  probability  (Green) 

Partial  evidence 

P  =  0.02  -  0.2;  average  0.11 

Very  Low  probability  (Blue) 

Strong  and  externally 
validated  evidence 

P  =  0  -  0.02;  average  0.01 

Again,  using  Question  1.1(a)  from  Figure  3  as  an  example  analyzing  a  C4ISR  system  with  several 
mission-critical  KPPs,  then  a  lack  of  evidence  (from  analysis  of  current-system  shortfalls  and/or  the  use 
of  operational  scenarios  and  prototypes)  that  its  “KPPs  had  been  identified  at  Milestone  A  in  clear, 
comprehensive,  concise  terms  that  are  understandable  to  the  users  of  the  system”  would  result  in  a  High 
risk  probability,  while  strong  and  externally  validated  evidence  would  result  in  a  Very  Low  risk 
probability. 

Using  the  average  probability  and  impact  values  from  Table  7  and  Table  8,  the  relative  Risk  Exposure  = 
P(Risk)  *  Size(Risk)  implied  by  the  ratings  is  presented  in  Table  9. 


Table  9.  Average  risk  exposure  calculation 


Impact  //  Probability 

Very  Low 

Low 

Medium 

High 

Critical 

0.7 

7.7 

21 

49 

Significant 

0.3 

3.3 

9 

21 

Moderate 

0.11 

1.21 

3.3 

7.7 

Little-No  Impact 

0.01 

0.11 

0.3 

0.7 
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The  prototype  tool  provides  a  customizable  mapping  of  each  impact/probability  pair  to  a  color-coded 
risk  exposure,  based  on  the  above  table.  For  each  question,  the  risk  exposure  level  is  determined  by  the 
combination  of  risk  impact  and  risk  probability,  and  a  corresponding  risk  exposure  color-coding  is 
selected,  which  ranges  from  red  for  the  highest  risk  exposure  to  green  for  the  lowest.  Figure  4  provides 
an  example  of  this  mapping  from  the  prototype  tool,  and  illustrates  the  resulting  risk  exposure  matrix  for 
the  selected  mapping. 


Impact  p(Risk)  Index  Color  Impact 


Figure  4.  Impact/risk  mapping  to  risk  exposure 


The  current  tool  assigns  the  highest  (red)  risk  exposure  for  the  (Impact,  Probability)  combinations  of 
(Critical,  Medium),  (Critical,  High),  and  (Significant,  High).  It  assigns  a  medium-high  (orange)  risk 
exposure  for  the  (Impact,  Probability)  combinations  of  (Critical,  Low),  (Significant,  Medium),  and 
(Moderate,  Medium).  A  medium  (yellow)  risk  exposure  is  assigned  for  the  (Impact,  Probability) 
combinations  of  (Significant,  Low),  and  (Moderate,  Medium).  A  medium-low  (green)  risk  exposure 
results  from  the  (Impact,  Probability)  combinations  of  (Critical,  Very  Low),  (Moderate,  Low),  and 
(Little-No,  High).  Finally,  all  remaining  combinations  involving  Little-No  impact  or  Very  Low 
probability  are  assigned  the  lowest  risk  exposure  (green). 

As  seen  in  Figure  3,  the  risk  exposure  resulting  from  scoring  the  impact  and  risk  of  each  question  is 
presented  in  the  leftmost  column.  Based  on  suggestions  from  workshop  participants,  the  current  version 
of  the  tool  assigns  the  highest  risk  exposure  level  achieved  by  any  of  the  questions  in  a  CSF  as  the  risk 
exposure  for  the  overall  CSF.  This  maximum  risk  exposure  presented  in  the  rightmost  column  for  the 
CSF.  This  rating  method  has  the  advantages  of  being  simple  and  conservative,  but  might  raise  questions 
if,  for  example,  CSF  1 . 1  were  given  a  red  risk  exposure  level  for  one  red  and  four  greens,  and  a  yellow 
risk  exposure  level  for  five  yellows.  Experience  from  piloting  of  the  tool  has  suggested  refinements  to 
this  approach,  discussed  later  in  this  report. 


3.2  SE  Competency  Assessment  Tool 

The  systems  engineering  competency  assessment  tool  (SECAT),  like  the  SEP  AT  tool  described  above, 
is  a  prototype  expression  of  the  framework  for  evaluating  SE  competency.  Although  the  framework  is 
believed  to  be  complete,  the  question  set  supporting  the  framework  is  not  based  on  a  coverage  matrix 
analysis,  but  largely  based  on  the  SEP  AT  framework  plus  the  additional  goal  and  five  CSFs  addressing 
Professional  and  Interpersonal  Skills.  SECAT’s  concise,  project-tailorable  content  complements  other 
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more  detailed  SE  personnel  competency  frameworks  that  focus  more  on  SE  personnel  certification  or 
organizational  skills  coverage. 

The  general  design  of  the  SECAT  prototype  is  identical  to  that  of  SEP  AT — an  Excel-based  spreadsheet. 
The  SECAT  impact  and  evidence  scales  are  the  same  as  SEP  AT,  except  that  the  ratings  for  evidence 
score  the  relative  experience  and  competency  of  the  team  needed  to  fulfill  the  SE  function.  This  scale 
reflects  the  team’s  composite  experience  and  competency  level  with  respect  to  its  relevant  systems 
engineering  technical,  professional,  and  applications  domain  knowledge,  skills,  and  abilities.  Similar  to 
SEP  AT,  the  SECAT  evidence  ratings  range  from  a  High  probability  of  an  unsuccessful  outcome  in 
performing  the  SE  task  in  question  (red;  no  relevant  experience  and  competency),  through  Medium 
probability  (yellow;  some  relevant  experience  and  competency)  and  Low  probability  (green;  good 
relevant  experience  and  competency),  to  very  low  probability  of  an  unsuccessful  outcome  (blue;  expert 
relevant  experience  and  competency). 

The  following  sections  describe  and  use  the  SEP  AT  tool  as  an  example,  though  the  discussion  applies 
equally  to  the  SECAT  tool.  It  is  expected  that  the  SECAT  framework  will  continue  to  evolve  and 
mature  in  future  research. 


4.  Description  and  Usage  of  Prototype  Tools 

The  prototype  Systems  Engineering  Performance  Assessment  Tool  (SEP AT)  and  Systems  Engineering 
Competence  Assessment  Tool  (SECAT)  tools  are  extendable  Excel  spreadsheets  organized  around  a 
framework  of  systems  engineering  goals,  contributing  critical  success  factor  questions,  and  detailed 
metric  questions.  Each  question  can  be  prioritized  for  relevance  to  the  particular  systems  engineering 
effort,  and  assessed  with  respect  to  the  degree  of  evidence  that  it  is  being  well  addressed. 

The  tools  are  intended  for  use  at  discrete  assessment  points  during  a  project’s  lifetime.  For  example,  the 
tools  might  be  used  to  review  SE  plans  or  preparations  for  major  milestone  reviews  to  assess  any 
shortfalls  in  SE  effectiveness.  The  SEP  AT  tool  addresses  the  evidence  of  thoroughness  with  which  core 
systems  engineering  functions  are  being  performed.  The  SECAT  tool  assesses  the  evidence  of  whether 
sufficient  SE  team  personnel  competence  is  in  place  to  carry  out  the  functions.  Both  tools  treat  a 
shortfall  in  evidence  as  an  increased  risk  probability.  This  probability,  multiplied  by  the  relative  impact 
of  the  item  on  project  success,  produces  a  risk  exposure  quantity,  color-coded  for  identification  of  the 
risk  levels  of  SE  effectiveness  items. 

The  primary  objective  of  piloting  the  prototype  tools  is  to  determine  the  utility  of  the  evaluation 
frameworks  at  various  points  in  a  project's  life  cycle,  across  different  acquirers  and  application  domains. 
Data  of  interest  from  piloting  these  tools  includes  both  the  cost  in  effort  required  to  perform  the 
assessments,  and  the  value  obtained  from  performing  them.  It  is  not  an  objective  of  the  pilot  evaluations 
for  evaluators  to  externally  disclose  shortfalls  or  risks  in  the  projects  assessed,  although  the  researchers 
are  seeking  information  on  the  effects  of  using  the  tools. 


4.1  Tool  Overview 

The  SEP  AT  and  SECAT  tools  are  Excel-based  prototypes.  The  tools  were  created  in  Excel  2007.  Users 
with  Excel  2003  will  have  the  same  functionality,  but  the  risk  exposure  color-coding  does  not  function. 
Macros  must  be  enabled  for  functionality.  The  SECAT  competency  evaluation  tool  operates  identically 
to  the  SEP  AT  tool  described  below,  though  the  critical  success  factors  and  evaluation  questions  differ. 
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Each  tool  identifies  high-level  Goals  that  must  be  met,  and  provides  four  or  five  Critical  Success  Factors 
that  support  each  goal.  Questions  then  explore  whether  the  critical  success  factors  are  being  met.  Each 
question  is  evaluated  against  two  separate  scales:  evidence  and  impact.  Less  evidence  is  equated  to 
higher  risk.  An  impact  rating  for  each  question  allows  the  evaluator  to  adjust  the  weighting  of  the 
question  for  that  particular  project. 

ft  is  recommended  that  the  impact  rating  and  evidence  scores  be  determined  by  independent  reviewers. 
For  instance,  the  project  or  program  manager  or  their  designate  might  provide  the  impact  ratings,  and 
project  chief  engineer  or  chief  systems  engineer  or  their  designate  might  provide  the  evidence  ratings. 

Figure  5  illustrates  the  rating  scale  for  impact  and  evidence  on  each  question.  In  the  leftmost  set  of 
selections,  the  evaluator  selects  an  appropriate  weighting  for  the  impact,  ranging  from  critical  impact 
(red)  to  little-no  impact  (gray).  Similarly,  the  rightmost  selection  set  indicates  the  degree  of  evidence 
available  that  supports  the  evaluation  of  each  question,  where  red  implies  little-or-no  evidence  has  been 
found  to  support  the  conjecture,  and  blue  implies  that  independent  experts  have  validated  the  evidence. 
Users  make  selections  by  clicking  on  the  appropriately  colored  boxes  for  each  question. 


Figure  5.  Detail  of  impact  and  evidence  rating 


As  seen  in  Figure  6,  the  impact  and  evidence  scores  for  each  critical  success  factor  are  rolled  up  into  an 
overall  risk  exposure,  which  again  is  represented  as  a  simple  red-orange-yellow-light  green-green 
indicator  (for  Excel  2003  users,  risk  exposure  is  5,  4,  3,  2,  1,  respectively).  The  overall  risk  exposure  is 
the  maximum  of  the  risk  exposures  determined  from  the  responses  to  the  individual  questions  that 
support  each  critical  success  factor.  The  “rationale”  column  may  be  used  to  record  the  source  of 
evidence  for  later  review.  The  “reset”  button  clears  the  impact  and  evidence  ratings  for  the  entire 
document. 
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Figure  6.  Detail  of  risk  exposure  rollup  by  CSF 


Risk  exposure  (RE)  is  calculated  by  multiplying  the  risk  impact  by  the  probability  of  risk  (exposure  = 
impact  *  p(risk)).  Since  the  impact  and  probability  of  risk  are  represented  here  as  discrete  quantities, 
however,  a  different  approach  was  used  to  determine  the  risk  exposure.  Figure  7  is  an  excerpt  from  the 
“RE  Map”  tab  of  the  SEP  AT  and  SECAT  tool  spreadsheets.  On  this  tab,  each  combination  of  impact 
and  p(risk) — where  zero  represents  little-no  impact/little-no  risk,  and  three  represents  critical 
impact/high  risk,  as  shown  by  the  “Scale” — may  be  assigned  a  value  from  one  (green)  to  five  (red)  by 
filling  in  the  “Color”  column.  The  risk  exposure  matrix  resulting  from  these  choices  is  automatically 
shown  on  the  right,  in  a  format  similar  to  the  five-by-five  representation  commonly  used  in  risk  analysis. 
In  this  example,  the  combination  of  “no  impact”  (impact=0)  and  “independently  validated  evidence” 
(p(Risk)=3)  result  in  a  medium-low  (light-green=2)  risk  exposure.  The  values  in  the  “Color”  column 
may  be  altered  to  suit  the  needs  of  the  program  being  evaluated. 


Figure  7.  Detail  of  risk  exposure  mapping 
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4.2  Tool  Tailoring  and  Extension 


Being  developed  in  Excel,  the  prototype  tools  are  relatively  simple  to  tailor  and  extend,  to  explore 
additional  framework  concepts.  Questions  can  be  added  to  a  CSF  by  copying  and  pasting  an  existing 
question  and  modifying  it.  The  formulas  for  computing  risk  exposure  are  straightforward,  and  can  be 
similarly  copied  and  adjusted  to  refer  to  the  new  questions. 

The  risk  exposure  mapping  itself  is  designed  to  be  configurable  by  the  individual  project.  Using  the 
“RE  Mapping”  tab,  as  described  in  Figure  7  above,  combinations  of  risk  impact  and  probability  can  be 
mapped  to  different  risk  exposure  ratings,  simply  by  modifying  the  RE  numbers  in  the  “Color”  column. 

Based  on  feedback  from  the  pilot  evaluations,  it  may  be  desirable  in  the  future  to  modify  the  EM 
frameworks  to  be  less  specific  to  DoD  terminology  and  milestones.  Although  the  exact  phrasing  of 
goals,  CSFs,  and  questions  is  a  topic  for  further  research,  the  tools  are  easily  modifiable  to  reflect  the 
desired  changes. 

There  is  some  interest  in  adapting  the  frameworks  to  address  the  specific  needs  of  particular  application 
domains.  For  example,  the  SE  needs  for  engineering  secure  systems  are  more  specific  than  the  general 
SE  needs  expressed  in  the  present  framework.  Similarly,  organizations  focused  on  ground,  sea,  air,  and 
space  missions  would  benefit  from  domain-specific  extensions.  Future  research  intends  to  explore  how 
the  framework  might  be  adapted  to  such  specific  needs,  and  to  adapt  the  prototype  tools  to  enable 
piloting  of  these  future  efforts. 


5.  Summary  of  Framework  and  Tool  Evaluations 

The  researchers  solicited  pilot  evaluations  of  the  EM  performance  and  competency  frameworks,  using 
the  prototype  SEPAT  and  SECAT  tools,  from  industry,  government  agencies,  and  academic 
participants.  Because  the  task  re-scoping  permitted  only  a  single  round  of  piloting,  these  initial 
evaluations  were  conducted  against  historical  projects  and  case  studies.  In  addition,  the  University  of 
Maryland  (UMD)  Fraunhofer  Center  (FC)  began  preliminary  evaluations  against  the  Systemic  Analysis 
Database  (SADB),  compiled  by  OUSD  (AT&L).  This  evaluation  approach  allowed  analysis  of  the 
effectiveness  of  the  frameworks  with  respect  to  historical  success  and  failures  of  the  subject  projects. 

The  tools  were  successfully  piloted  against  five  DoD  projects,  one  NASA  project,  and  one  commercial 
project.  They  were  also  analyzed  by  two  industrially-experienced  colleagues  against  detailed  case 
studies  of  a  number  of  DoD  and  commercial  projects.  The  application  domains  piloted  included  space, 
medical  systems,  logistics,  and  systems-of-systems.  Results  of  the  pilot  evaluations  were  reported 
through  a  web-based  survey  tool  and  detailed  follow-up  interviews,  while  the  case  study  evaluations 
were  reported  through  detailed  comments  from  the  reviewers. 

Evaluations  were  generally  positive,  and  the  frameworks  were  found  to  be  useful  across  all  project 
phases  except  Production,  and  against  all  systems  types  except  “legacy  development.”  The  consensus  of 
reviewers  was  that  the  frameworks  would  be  most  useful  in  the  System  Development  &  Demonstration 
(SDD)  phase,  and  generally  more  useful  in  early  phases  than  later.  It  was  noted,  however,  that  in 
systems  developed  using  evolutionary  strategies,  such  “early”  phases  recur  throughout  the  development 
cycle,  extending  the  usefulness  of  the  frameworks.  The  evaluations  were  reported  to  take  2-5  hours  to 
complete  for  persons  familiar  with  the  projects,  with  materials  that  were  readily  at  hand. 
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Some  concerns  reported,  particularly  by  the  NASA  reviewers,  were  that  the  frameworks  were  too  DoD- 
centric  in  their  terminology,  milestones,  and  acquisition  frameworks.  These  reviewers  were  sufficiently 
familiar  with  DoD  processes  to  allow  analysis  of  the  NASA  projects,  but  suggested  that  the  CSFs  and 
questions  be  modified  to  be  more  general.  In  reviewing  case  study  material,  some  evaluators  reported 
that  the  EM  framework  was  not  specific  to  any  particular  problem  domain.  Finally,  several  evaluators 
reported  that  the  frameworks  generated  too  many  high-risk  findings,  which  might  make  the  results  too 
overwhelming  to  take  action. 

These  concerns  were  partially  mitigated  with  suggestions  from  the  evaluators.  Although  the  general 
nature  of  the  EM  frameworks  was  intentional  in  this  portion  of  the  research,  it  suggests  that  tailoring 
might  be  useful  to  uncover  domain-specific  shortcomings  in  SE  functions.  Two  approaches  might  be 
used  with  respect  to  the  DoD-specific  nature  of  the  frameworks:  the  frameworks  might  be  edited  to  use 
more  generic  terms,  or  new  versions  tailored  to  other  agencies  and  domains.  With  respect  to  the 
frameworks  uncovering  too  many  high-risk  findings,  the  impact  scales  have  been  adjusted  to  make  the 
adjectives  better  correspond  to  the  quantitative  impacts  (Critical-Significant-Moderate-Little  or  No  vs. 
High-Medium-Low-No  impact),  and  a  longer  risk  exposure  scale  developed  to  allow  more  nuanced 
results. 

Details  regarding  use  of  the  SEP  AT  (performance)  tool  are  as  follows.  It  was  found  to  be  particularly 
useful  in  SDD,  somewhat  useful  in  early  phases,  and  less  useful  in  later  phases.  This  result  is  somewhat 
expected,  given  the  “early-phase”  emphasis  of  the  materials  used  as  sources  for  the  frameworks,  and 
suggests  that  further  research  might  examine  the  SE  strategies  that  are  important  in  later  life-cycle 
phases,  such  as  testing  and  configuration  management.  The  evaluators  also  report  that  the  SEP  AT 
effectiveness  ranges  from  very  effective  to  somewhat  effective,  with  the  majority  reporting  it  as 
effective.  Only  analysis  of  one  DoD  legacy  system  project  reported  the  tool  as  ineffective,  which 
suggests  that  the  issues  facing  such  projects  may  need  to  be  examined  more  closely.  Even  with  respect 
to  DoD  systems,  some  evaluators  reported  issues  with  the  tenninology  used,  which  might  be  cleared  up 
with  careful  editing. 

With  respect  to  the  SECAT  (competency),  the  evaluators  also  found  the  tool  most  useful  in  earlier  life- 
cycle  phases,  rather  than  later.  The  usefulness  was  rated  between  effective  and  somewhat  effective. 
Although  the  SECAT  is  less  well  developed  than  the  SEP  AT,  one  reviewer  noted  that  the  shortfalls 
noted  by  the  SECAT  might  well  be  used  to  help  justify  and  explain  the  need  for  stronger  SE  capabilities 
to  program  management  and  acquirers.  Another  reviewer  observed  that  the  choice  of  person  perfonning 
the  competency  evaluation  was  critical,  as  it  is  difficult  for  a  non-technical  evaluator  to  judge 
competency.  Finally,  one  comment  that  is  more  a  judgment  of  state  of  practice  is  that,  “to  be  effective, 
program  management  must  have  some  control  over  who  is  assigned.” 

Several  reviewers  comment  that  the  simple  red,  yellow,  green,  and  blue/gray  choices  for  impact  and 
evidence  ratings  might  be  too  granular.  However,  the  initial  rating  scales  used  a  1-5  range,  which 
workshop  participants  judged  as  too  complex.  The  resulting  granularity  of  risk  exposure  (RE)  results 
might  be  mitigated  using  several  techniques.  The  RE  calculation  has  been  broadened  to  allow  medium- 
high  (orange)  and  medium-low  (light  green)  results,  which  reduces  the  clumping  of  all  results  into 
medium  (yellow)  and  high  (red).  The  impact  scale  has  also  been  clarified  to  make  the  lowest  choice 
“little  or  no”  impact,  rather  than  “no”  impact.  Since  the  prototype  tools  presently  use  the  maximum  RE 
of  all  questions  in  a  CSF,  another  suggestion  was  to  provide  a  count  of  the  number  of  REs  at  this 
maximum  level,  as  a  “red  (4)”  is  clearly  worse  than  a  “red  (1).”  All  but  the  last  of  these  suggestions 
have  already  been  retrofitted  into  the  SEP  AT  and  SECAT  tools,  although  the  changes  have  not  yet  been 
re-piloted. 
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The  evaluations  indicate  that  the  concept  of  “evidence-based”  information  is  not  well  understood,  and 
may  require  further  explanation  and  example.  For  example,  one  reviewer  asks  specifically  what 
constitutes  “sufficient”  evidence.  One  answer  is  evidence  that  an  independent,  expert  reviewer  would 
find  sufficient  as  a  compelling  and  objectively  verifiable  argument  of  feasibility.  This  is  a  slight 
modification  of  the  original  definition,  where  the  word  “external”  was  used  rather  than  “independent,” 
as  one  reviewer  observed  that  for  some  domains,  external  reviewers  would  not  have  sufficient  context  to 
perform  knowledgeable  assessments.  It  might  also  be  necessary  to  supplement  the  framework  question 
set  with  material  that  describes  the  intended  goal  for  each  question,  so  that  evidence  can  be  judged 
against  this  goal. 

In  summary,  the  framework  and  prototype  tools  have  been  shown  to  be  largely  efficacious  for  pilot 
projects  done  by  familiar  experts  in  a  relatively  short  time,  in  identifying  characteristics  of  SE  efforts 
that,  inadequately  perfonned,  will  likely  lead  to  difficulties  with  the  program.  It  remains  to  demonstrate 
how  well  the  framework  and  tools  will  perform  on  in-process  MDAPs  with  multiple  missions, 
performers,  and  independent  expert  assessors. 


C.  DoD  MDAP  SE  EM  Concepts  of  Operation 


The  SEPAT  and  SEC  AT  framework  and  tools  provide  a  way  for  MDAPs  (and  other  projects)  to  identify 
the  major  sources  of  program  risk  due  to  SE  shortfalls.  This  section  provides  concepts  of  operation  for 
applying  the  tools  at  major  milestones,  and  at  other  points  where  other  SE  EMs  such  as  the  INCOSE 
Leading  Indicators  have  identified  likely  problem  situations  and  need  further  understanding  of  the 
problem  sources  and  their  relative  degrees  of  risk. 

Section  C.l  establishes  the  primary  criteria  for  satisfactory  program  evidence,  and  provides  a  vehicle  for 
capturing  the  evidence  called  a  Feasibility  Evidence  Description.  Section  C.2  provides  representative 
MDAP  operational  scenarios  that  show  how  the  detennination  of  SEPAT  and  SECAT  impact  priorities 
can  be  done  collaboratively  by  a  program’s  sponsors  and  performers  at  various  life  cycle  points, 
enabling  a  cooperative  rather  than  an  adversarial  approach  to  system  definition  and  development. 
Section  C.3  shows  how  the  use  of  feasibility  evidence  as  a  first-class  deliverable  enables  programs  to 
plan  and  control  progress  toward  successful  passage  of  SEPAT-  and  SECAT-based  reviews.  Section 
C.4  describes  a  process  for  conducting  such  reviews,  and  summarizes  successful  experiences  in 
applying  such  processes. 


1.  Evidence  Criteria  and  Review  Milestone  Usage 

Having  shown  in  Section  A. 3  that  the  regions  of  high  payoff  for  evidence-based  specifications,  plans, 
assessments,  and  reviews  are  extensive  and  enterprise-critical,  it  is  now  important  to  define  the  criteria 
for  the  evidence  that  will  be  associated  with  the  system  development  specifications  and  plans,  and 
reviewed  via  the  SEPAT  and  SECAT.  The  criteria  are  extensions  to  those  for  the  Anchor  Point  (AP) 
milestones  defined  in  [7]  and  adopted  by  the  Rational  Unified  Process  (RUP)  [18], [25].  The  extensions 
have  been  incorporated  into  the  recently  developed  Incremental  Commitment  Model  (ICM);  details  of 
this  model  can  be  found  in  [10], [23].  Also,  comparable  benefits  can  be  obtained  by  adding  such  criteria, 
processes,  and  incentive  structures  to  traditional  acquisition  methods  and  reviews. 
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The  evidence  criteria  are  embodied  in  a  Feasibility  Evidence  Description  (FED).  It  includes  evidence 
provided  by  the  developer  and  validated  by  independent  experts  that,  if  the  system  is  built  to  the 
specified  architecture  it  will: 

1 .  Satisfy  the  specified  operational  concept  and  requirements,  including  capability,  interfaces,  level  of 
service,  and  evolution 

2.  Be  buildable  within  the  budgets  and  schedules  in  the  plan 

3.  Generate  a  viable  return  on  investment 

4.  Generate  satisfactory  outcomes  for  all  of  the  success-critical  stakeholders 

5.  Identify  shortfalls  in  evidence  as  risks,  and  cover  them  with  risk  mitigation  plans 

A  FED  does  not  assess  a  single  sequentially  developed  system  definition  element,  but  the  consistency, 
compatibility,  and  feasibility  of  several  concurrently  engineered  elements.  To  make  this  concurrency 
work,  a  set  of  Anchor  Point  milestone  reviews  are  performed  to  ensure  that  the  many  concurrent 
activities  are  synchronized,  stabilized,  and  risk-assessed  at  the  end  of  each  phase.  Each  of  these  reviews 
is  focused  on  developer-produced  and  expert-validated  evidence,  documented  in  the  FED  (or  by  pointers 
to  the  results  of  feasibility  analyses),  to  help  the  system’s  success-critical  stakeholders  determine 
whether  to  proceed  into  the  next  level  of  commitment.  Hence,  they  are  called  Commitment  Reviews. 

The  FED  is  based  on  evidence  from  simulations,  models,  or  experiments  with  planned  technologies  and 
increasingly  detailed  analysis  of  development  approaches  and  project  productivity  rates.  The 
parameters  used  in  the  analyses  should  be  based  on  measured  component  perfonnance  or  on  historical 
data  showing  relevant  past  perfonnance,  cost  estimation  accuracy,  and  actual  developer  productivity 
rates. 

As  detennined  by  independent  experts  using  the  SEPAT  and  SECAT  questions,  a  shortfall  in  feasibility 
evidence  indicates  a  level  of  program  execution  uncertainty  and  a  source  of  program  risk.  It  is  often  not 
possible  to  fully  resolve  all  risks  at  a  given  point  in  the  development  cycle,  but  known,  unresolved  risks 
need  to  be  identified  and  covered  by  risk  management  plans,  including  the  necessary  staffing  and 
funding  to  address  them.  The  nature  of  the  evidence  shortfalls,  the  strength  and  affordability  of  the  risk 
management  plans,  and  the  stakeholders’  degrees  of  risk  acceptance  or  avoidance  will  determine  their 
willingness  to  commit  the  necessary  resources  to  proceed.  A  program  with  risks  is  not  necessarily  bad, 
particularly  if  it  has  strong  risk  management  plans.  A  program  with  no  risks  may  be  high  on 
achievability,  but  low  on  ability  to  produce  a  timely  payoff  or  a  competitive  advantage. 

A  FED  needs  to  be  more  than  just  traceability  matrices  and  PowerPoint  charts.  Evidence  can  include 
results  of: 

•  Prototypes:  of  networks,  robots,  user  interfaces,  COTS  interoperability 

•  Benchmarks:  for  performance,  scalability,  accuracy 

•  Exercises:  for  mission  performance,  interoperability,  security 

•  Models:  for  cost,  schedule,  performance,  reliability;  tradeoffs 

•  Simulations:  for  mission  scalability,  performance,  reliability 
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•  Early  working  versions:  of  infrastructure,  data  fusion,  legacy  compatibility 

•  Previous  experience 

•  Combinations  of  the  above 

Not  only  does  the  evidence  need  to  be  produced,  but  it  needs  to  be  validated  by  independent  experts. 
These  experts  need  to  determine  the  realism  of  assumptions,  the  representativeness  of  scenarios,  the 
thoroughness  of  analysis,  and  the  coverage  of  key  off-nominal  conditions.  At  least  one  DoD  MDAP  has 
explicitly  included  a  FED  Data  Item  Description  in  its  list  of  deliverables. 


2.  Use  of  the  SERC  SE  EM  Capabilities  in  Evidence-Based  SE 

This  section  shows  how  the  EMs  can  be  used  to  reach  sponsor-performer  consensus  on  the  relative 
impact  of  each  EM  upon  the  project  outcome,  and  of  the  resources  required  to  achieve  it.  These  then 
serve  as  a  consensus-based  set  of  criteria  that  will  be  used  to  measure  evidence  of  the  project's  SE 
effectiveness,  which  is  then  reinforced  by  becoming  a  significant  determinant  of  the  performer's  award 
fee.  Shortfalls  in  evidence  are  uncertainties  or  probabilities  of  loss,  which  when  multiplied  by  the 
relative  impact  or  size  of  loss,  become  measures  of  project  risk.  These  then  require  risk  management 
plans,  employing  the  major  risk  mitigation  options  of  buying  information,  risk  avoidance,  risk  transfer, 
risk  reduction,  or  risk  acceptance.  Three  early-SE  scenarios  are  provided: 

1 .  At  Milestone  A:  Milestone  Decision  Authority  (MDA)  Review  of  Acquirer  Plans 

2.  Contract  Negotiation:  MDAP  Acquirer  and  Developer 

3.  Project  Execution:  MDAP  Developer  Manager  and  Performers. 

2.1  Scenario  1:  MDA  Review  of  Acquirer  Plans  at  Milestone  A 

Step  1 .  The  Acquirer  submits  a  proposed  acquisition  plan  to  the  MDA,  along  with  the  SEP  AT 
and  SECAT  evidence  ratings  and  risk  mitigation  approaches. 

Step  2.  The  MDA  has  independent  experts  review  the  SEP  AT  and  SECAT  ratings.  A  major 
finding  is  that  no  Analysis  of  Alternatives  has  been  performed.  Only  one  alternative  has  been 
analyzed,  but  the  relative  risk  shown  in  SEP  AT  is  low  because  the  Acquirer  has  assigned  it  a 
Little  or  No  Impact  rating. 

Step  3,  Case  1.  The  Acquirer  agrees  that  the  capability  needed  is  critical,  but  that  it  is  needed 
quickly  for  a  relatively  unique  and  short-term  threat,  and  that  evidence  is  available  that  a 
solution  involving  Alternative  A  will  be  sufficient.  The  MDA  concurs,  and  gives  approval  to 
rapidly  proceed  with  Alternative  A. 

Step  3,  Case  2.  The  Acquirer  states  that  a  Defense  Advanced  Research  Projects  Agency 
(DARPA)  demo  of  the  Alternative  X  technology  has  shown  proof  of  principle  of  its  feasibility, 
and  that  all  that  is  needed  is  to  implement  Alternative  X  for  the  general  case.  The  independent 
experts  conclude  that  the  proof  of  principle  demo  provided  no  evidence  of  the  solution's 
scalability  or  ability  to  work  in  degraded  battle  conditions.  The  MDA  does  not  give  an  approval 
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to  proceed  with  Alternative  X,  but  directs  the  Acquirer  to  resubmit  using  a  Competitive 
Prototyping  acquisition  approach. 


In  each  case,  the  MDA  and  Acquirer  agree  on  an  acceptable  approach.  In  Case  1,  the  outcome  is  a 
timely  and  acceptable  solution.  In  Case  2,  Competitive  Prototyping  is  used  as  a  way  of  buying 
information  to  reduce  risk.  At  the  end  of  the  first  round  of  prototyping,  no  competitor  may  be  able  to 
develop  a  scalable  and  robust  solution,  and  the  acquisition  can  be  deferred  until  the  technology  is  more 
mature.  Or  it  may  be  that  one  or  more  competitors  have  produced  scalable  and  robust  solutions,  and 
another  round  of  prototyping  will  determine  the  best  approach  and  supplier. 


2.2  Scenario  2:  Acquirer  and  Winning-Prototype  Developer  Contract  Negotiation 

Step  1.  The  Acquirer  specifies  SEP  AT  Critical  Success  Factor  1.2(d),  “Have  the  claimed  quality 
of  service  guarantees  been  validated?”  to  have  Critical  impact,  and  thus  to  be  a  major 
determinant  of  the  Performer’s  award  fee  at  each  review  milestone. 

Step  2.  The  Developer  is  the  winning  competitive  prototyping  developer,  and  clearly  the 
performer  of  choice.  Their  response  is,  “We  agree  that  quality  of  service  evidence  is  critical,  as 
is  addressing  it  before  committing  to  functional  requirements.  But  we  have  tied  our  plans  and 
budgets  to  the  first  milestone  in  your  contract,  a  System  Functional  Requirements  Review  that 
ties  our  progress  payments  and  award  fees  to  just  specifying  functionality.  If  you  agree  that 
early  QoS  evidence  is  critical,  we  need  to  find  a  way  to  emphasize  this  in  the  contract.” 

Step  3.  The  Acquirer  responds,  “Thanks.  That  legacy  contract  clearly  undercuts  our  intent  to  do 
evidence-based  concurrent  engineering,  and  sets  us  up  for  late  overruns.  We’ll  redo  it  and  your 
SE  plans  and  budgets.  Next  time,  we’ll  address  contracting  compatibility  earlier.” 

2.3  Scenario  3:  Acquirer  and  Developer  Contract  Performance 

Having  agreed  on  a  revised  contract  and  increased  early-SE  budget  and  schedule,  justified  by  the 
rework-avoidance  business  case  in  Section  A. 3,  the  Acquirer  and  Developer  use  Figure  8  as  a  basis  for 
proceeding.  Since  the  system  is  not  a  quick-response  development  for  a  relatively  unique  and  short¬ 
term  threat,  the  opportunistic  development  branch  is  not  chosen.  The  evaluation  of  SE  plans  and 
staffing  results  in  the  independent  evaluators  indicating  that  the  plans  and  staffing  are  sound,  but  also 
identifying  an  available  mission  simulator  that  can  increase  the  cost-effectiveness  of  the  evidence 
generation,  which  the  Developer  incorporates  into  the  plans. 

During  development,  the  project  uses  the  INCOSE  Leading  Indicators  capability  to  flag  any  progress 
indicators  that  exceed  a  set  of  control  limits  agreed  upon  by  the  Acquirer  and  Developer.  At  some  point, 
a  Leading  Indicator  shows  that  progress  is  behind  schedule  in  modifying  the  mission  simulator.  A 
specialized  SEP  AT  assessment  of  the  simulator  subproject  finds  that  difficulties  have  been  encountered 
in  getting  the  simulator  to  provide  evidence  that  the  system  can  handle  some  off-nominal  threat 
scenarios  that  the  simulator  was  not  designed  to  handle.  A  quick  risk  mitigation  plan  is  developed  to 
bring  aboard  some  experts  to  rework  the  simulator  in  time  to  be  used  for  the  off-nominal  scenarios. 
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Figure  8.  Project  SysE  EM  Operational  Concept  (for  each  stage  of  system  definition  and 

development) 


3.  FED  Development  Process  Framework 

The  most  important  characteristic  of  evidence-based  system  specifications  and  plans  is  that: 

•  If  the  evidence  does  not  accompany  the  specifications  and  plans,  the  specifications  and  plans  are 
incomplete. 

Thus,  event-based  reviews,  where  the  event  is  defined  as  production  of  the  specifications  and  plans, 
need  to  be  replaced  by  evidence-based  reviews. 

This  does  not  mean  that  the  project  needs  to  spend  large  amounts  of  effort  in  documenting  evidence  of 
the  feasibility  of  a  simple  system.  The  appropriate  level  of  detail  for  the  contents  of  the  FED  is  based  on 
the  perceived  risks  and  criticality  of  the  system  to  be  developed.  It  is  NOT  a  “one  size  fits  all”  process, 
but  rather  a  framework  to  help  developers  and  stakeholders  determine  the  appropriate  level  of  analysis 
and  evaluation.  As  with  reused  specifications  and  plans,  evidence  can  be  appropriately  reused.  If  a 
more  complex  system  than  the  one  being  reviewed  has  been  successfully  developed  by  the  same  team,  a 
pointer  to  the  previous  project’s  evidence  and  results  will  be  sufficient. 

Table  10  outlines  a  process  that  can  be  used  for  developing  feasibility  evidence.  The  process  clearly 
depends  on  having  the  appropriate  work  products  for  the  phase  (Step  A).  As  part  of  the  engineering 
work,  high-priority  feasibility  assurance  issues  are  identified  that  are  critical  to  the  success  of  the  system 
development  program  (Step  B).  These  are  the  issues  for  which  options  are  explored,  and  potentially 
viable  options  further  investigated  (Step  C).  Clearly,  these  and  the  later  steps  are  not  performed 
sequentially,  but  concurrently. 
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Table  10.  Steps  for  Developing  a  FED 


Step 

Description 

Examples/Detail 

A 

Develop  phase  work-products/artifacts 

For  a  Development  Commitment  Review  (DCR), 
this  would  include  the  system’s  operational 
concept,  prototypes,  requirements,  architecture,  life 
cycle  plans,  and  associated  assumptions 

B 

Detennine  most  critical  feasibility 
assurance  issues 

Issues  for  which  lack  of  feasibility  evidence  is 
program-critical 

C 

Evaluate  feasibility  assessment  options 

Cost-effectiveness;  necessary  tool,  data,  scenario 
availability 

D 

Select  options,  develop  feasibility 
assessment  plans 

The  list  of  options  at  the  end  of  Section  C.  1 
(prototypes,  benchmarks,  exercises,  etc)  is  a  good 
starting  point 

E 

Prepare  FED  assessment  plans  and 
earned  value  milestones 

The  plans  include  the  enablers  in  Step  G 

F 

Begin  monitoring  progress  with  respect 
to  plans 

Also  monitor  changes  to  the  project,  technology, 
and  objectives,  and  adapt  plans 

G 

Prepare  evidence-generation  enablers 

Assessment  criteria 

Parametric  models,  parameter  values,  bases  of 
estimate 

COTS  assessment  criteria  and  plans 

Benchmarking  candidates,  test  cases 
Prototypes/simulations,  evaluation  plans,  subjects, 
and  scenarios 

Instrumentation,  data  analysis  capabilities 

H 

Perform  pilot  assessments;  evaluate  and 
iterate  plans  and  enablers 

Short  bottom-line  summaries  and  pointers  to 
evidence  files  are  generally  sufficient 

I 

Assess  readiness  for  Commitment 
Review 

Shortfalls  identified  as  risks  and  covered  by  risk 
mitigation  plans 

Proceed  to  Commitment  Review  if  ready 

J 

Hold  Commitment  Review  when  ready; 
adjust  plans  based  on  review  outcomes 

See  Commitment  Review  process  overview  below. 

NOTE:  “Steps”  are  denoted  by  letters  rather  than  numbers  to  indicate  that  many  are  done 
concurrently. 

Since  the  preliminary  design  and  plans  are  incomplete  without  the  FED,  it  becomes  a  first-class  project 
deliverable.  This  implies  that  it  needs  a  plan  for  its  development,  and  that  each  task  in  the  plan  needs  to 
be  assigned  an  appropriate  earned  value.  If  possible,  the  earned  value  should  be  based  on  the  potential 
risk  exposure  costs,  not  the  perceived  available  budget. 

Besides  monitoring  progress  on  developing  the  system,  the  project  needs  to  monitor  progress  on 
developing  the  feasibility  evidence.  This  implies  applying  corrective  action  if  progress  falls  behind  the 
plans,  and  adapting  the  feasibility  evidence  development  plans  to  changes  in  the  project  objectives  and 
plans.  If  evidence  generation  is  going  to  be  complex,  it  is  generally  a  good  idea  to  perfonn  pilot 
assessments.  The  preparations  for  the  commitment  review  are  discussed  next. 
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1.  Commitment  Review  Process  Overview 


Figure  9  highlights  the  activities  that  need  to  be  perfonned  in  preparation  for  the  review,  the  actual 
review,  as  well  as  the  post-review  activities  and  follow-up.  The  entry  criteria  include  ensuring  that  the 
feasibility  evidence  preparation  has  been  successfully  tracking  its  earned  value  milestones.  The  inputs 
include  preparing  the  domain  extensions  to  the  core  SEP  AT  and  SECAT  framework  and  tools, 
identifying  committed  expert  reviewers  for  each  of  the  review  questions,  and  familiarizing  them  with  the 
SEPAT-SECAT  review  process. 

The  review  meeting  will  include  not  only  the  developer  SEs  and  the  expert  reviewers,  but  also  the 
stakeholder  upper-management  decisionmakers,  who  will  need  some  context-setting  before  the 
developer  responses  to  reviewer  issues  are  discussed.  The  review  exit  criteria  and  tasks  include  key 
stakeholder  concurrence  on  the  way  forward  and  commitment  to  support  the  next  phase,  as  well  as 
action  plans  and  risk  mitigation  plans  for  the  issues  identified. 


Figure  9.  Overview  of  Commitment  Review  Process 


4.1  Examples  of  Successful  Experiences  with  Evidence-Based  Reviews 

AT&T  and  its  spinoffs  (Telcordia,  Lucent,  Avaya,  and  regional  Bell  companies)  have  been  successfully 
using  versions  of  evidence-based  reviews  since  1988.  On  average,  there  has  been  a  10%  savings  per 
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reviewed  project,  with  substantially  larger  savings  on  a  few  reviewed  projects.  More  detail  on  their 
Architecture  Review  experience  is  in  [19]. 

The  million-line  TRW  CCPDS-R  project  summarized  in  [25]  by  Walker  Royce,  and  several  similar 
TRW  follow-on  projects  were  further  successful  examples.  Evidence-based  anchor  point  milestone 
reviews  are  also  an  integral  part  of  the  Rational  Software  Process,  with  many  successful 
implementations  [26],  although  a  good  many  RUP  applications  have  been  unable  to  succeed  because  of 
the  type  of  unaddressed  contractual  constraints  discussed  in  Section  2.2. 

The  highly  successful  Abbott  Laboratories’  next  generation  intravenous  infusion  pump  documented  in 
chapter  5  of  [23]  is  a  good  commercial  example  of  evidence-based  specifications  and  plans. 


D.  Pilot  SEPAT  and  SECAT  Evaluation  Processes  and  Lessons  Learned 

1.  Introduction 

SEPAT  and  SECAT  were  reviewed  by  NASA’s  Marshall  Space  Flight  Center  (MSFC)  to  provide  an 
external  assessment  of  the  potential  application  of  the  tools  in  a  non-DoD  development  environment  and 
to  provide  an  external  review  of  the  tools  by  product  developers  working  in  a  similar  complex  system 
environment.  The  review  team  was  comprised  a  senior  systems  engineer  and  senior  project  manager  at 
MSFC,  supported  by  two  researchers  from  The  University  of  Alabama  in  Huntsville  (UAH).  Data  from 
this  review  was  used  as  inputs  to  the  sponsor  workshops  and  in  the  development  of  proposals  for 
extensions  to  this  research  effort. 


2.  Methodology 

The  review  of  the  SEPAT  and  SECAT  tools  was  done  in  four  phases.  Phase  one  was  a  briefing  to  the 
assessment  team  on  the  research  initiatives,  a  review  of  the  SEPAT  and  SECAT  tools,  and  a  discussion 
on  how  the  review  data  would  be  collected.  Phase  two  was  an  initial  assessment  of  the  tools  by  a  senior 
systems  engineer.  Phase  three  was  a  detailed  assessment  of  the  tools  by  the  entire  team.  Phase  four  was 
the  reporting  of  the  assessment  results  by  interview  and  by  completing  a  web-based  survey. 

The  phase  one  briefing  was  a  ninety  minute  session  that  included  both  the  MSFC  and  UAH  team 
members.  It  was  held  at  MSFC.  The  briefing  included  an  overview  of  the  SERC  and  a  description  of 
the  specific  research  task  that  the  assessment  was  part  of.  A  review  of  the  data  collection  methods  was 
included.  The  MSFC  team  noted  that  they  would  prefer  to  discuss  their  review  findings  before 
completing  the  web-based  survey,  so  the  research  plan  was  modified  to  include  an  interview  in  phase 
four.  The  MSFC  team  was  instructed  to  contact  the  UAH  team  members  if  any  questions  on 
interpretation  of  the  tools  came  up  during  the  review. 

The  phase  two  initial  assessment  of  the  tool  was  done  by  the  MSFC  senior  systems  engineer.  The 
SEPAT  and  SECAT  were  provided  along  with  the  assessment  methodology  and  supporting  documents. 
The  systems  engineer  did  have  experience  at  multiple  government  agencies,  but  focused  the  assessment 
based  on  the  tools  potential  use  at  NASA.  Each  of  the  tools  was  reviewed  individually  and  a  written 
summary  of  comments  and  questions  was  provided. 


29 


The  phase  three  detailed  assessment  was  conducted  by  the  senior  systems  engineer  and  senior  project 
manager.  An  additional  junior  systems  engineer  was  at  the  meeting  but  was  not  significantly  involved 
in  the  review.  The  results  of  the  initial  review  were  discussed  and  specific  questions  on  interpretation  of 
some  questions  were  answered.  The  MSFC  team  did  contact  the  UAH  team  for  clarification  on  some 
questions.  A  summary  of  the  findings  was  then  prepared. 

The  phase  four  reporting  of  the  results  was  done  by  submitting  the  summary  of  findings  and  conducting 
a  telephone  interview  to  discuss  findings  of  the  review.  The  web-based  survey  was  then  completed  and 
a  copy  of  the  results  of  the  survey  was  reviewed  by  the  team  members. 

3.  Results 

The  assessment  was  done  to  provide  an  external  assessment  of  the  potential  application  of  the  tools  in  a 
non-DoD  development  environment  and  to  provide  an  external  review  of  the  tools  by  product 
developers  working  in  a  similar  complex  system  environment.  The  SEP  AT  and  SECAT  were  found  to 
be  generally  effective  in  supporting  an  aerospace  flight  hardware  program  at  NASA;  however, 
differences  in  the  models  used  for  project  development  and  terminology  differences  would  need  to  be 
addressed  before  the  model  would  be  applicable  for  general  use. 

The  SEPAT  and  SECAT  were  found  to  be  the  most  useful  in  concept  refinement,  system  development 
&  demonstration,  and  operations  &  support  phases.  The  tools  were  less  useful  in  the  technology 
development  and  production  &  deployment  phases.  The  assessment  team  noted  that  the  tools  could  be 
used  at  the  program,  project,  and  task  level.  Some  technology  development  activities,  such  as  science 
experiments,  may  not  find  the  tools  as  helpful  because  of  the  high  degree  of  process  tailoring.  The  team 
also  noted  that  there  were  differences  in  the  way  each  tool  presented  questions.  It  would  be  easier  for 
the  evaluators  if  the  tools  were  consistent  in  their  formats. 

The  evidence  required  to  complete  SEPAT  and  SECAT  was  generally  available.  It  was  noted  that  this 
could  be  highly  variable  and  is  dependent  in  part  on  the  experience  of  the  evaluators,  the  phase  the 
specific  program  is  in,  and  the  contracting  mechanisms  used.  Of  specific  note  was  the  experience  of  the 
evaluators  in  being  able  to  recognize  and  locate  relevant  artifacts  to  support  responses  to  specific 
questions  in  the  tools.  Accessing  artifacts  from  suppliers  could  be  of  concern  if  the  original  contracting 
documents  do  not  call  out  the  need  for  these. 

The  NASA  team  reported  approximately  five  labor  hours  was  required  by  their  team  members  to 
complete  each  of  the  tools.  It  was  noted  that  this  time  would  be  highly  variable,  dependent  on  the 
experience  of  the  assessment  team  members  and  the  phase  the  program  is  in.  Of  specific  concern  was 
the  need  to  adjust  the  tool  to  match  the  specific  uses  product  developer’s  model.  Having  an  external 
resource,  in  this  case  the  UAH  team,  to  provide  guidance  was  seen  as  valuable  to  the  effective  use  of  the 
tool. 

SEPAT  and  SECAT  were  reported  to  be  somewhat  effective  in  identifying  all  performance  risks.  The 
assessment  team  noted  that  it  can  be  difficult  to  isolate  the  performance  risk  portions  of  the  tool  from  the 
general  goals  and  questions,  and  nearly  all  the  questions  could  be  interpreted  as  having  an  effect  on 
performance  risk.  Understanding  the  purpose  of  the  tools  and  how  the  data  will  be  used  by  the  program 
management  team  was  noted  as  important  in  using  the  tools  effectively. 

The  team  reported  that  SEPAT  and  SECAT  were  assessed  to  be  somewhat  effective  in  helping  a 
program  team  use  an  evidence  based  approach  to  determining  performance  risks.  This  reported  result  is 
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of  note  because  the  tool  is  currently  structured  for  a  DoD  product  development  environment  and  does 
not  currently  have  tailoring  options  to  support  other  non-DoD  applications. 


4.  Lessons  Learned 

The  NASA  assessment  provided  several  lessons  and  opportunities  for  future  extensions  to  this  work. 
SEP  AT  and  SEC  AT,  in  their  current  fonn,  are  targeted  for  DoD  MDAPs.  The  trade-off  is  that  the  more 
specific  the  tools  are  the  more  effective  they  can  potentially  be  in  managing  program  performance  risks. 
This  increased  effectiveness  is  offset  by  the  lack  of  a  broad  application  of  the  tools  to  other,  non-DoD 
applications.  One  extension  of  this  research  would  be  to  develop  a  tailoring  option  for  the  tools  so  that 
individual  programs  could  adjust  the  scope  of  the  tools  to  meet  their  specific  program  needs.  This 
tailoring  could  be  done  based  on  the  requirements  called  out  in  the  program  SEP  or  SEMP.  Another 
extension  would  be  to  develop  a  specific  application  set  for  DoD  and  a  second  more  generic  set  for  other 
applications. 

The  assessment  team  also  noted  the  tradeoff  between  having  an  experienced  but  heavily  tasked  senior 
assessment  team  complete  the  tools  versus  a  less  experienced  but  available  junior  team.  Senior 
managers  will  want  to  be  familiar  with  the  tools  if  they  are  going  to  implement  the  results.  Completion 
of  the  tool  would  most  likely  be  delegated  to  the  systems  engineering  team  or  the  logistics  team  to 
complete.  Having  an  experienced  evaluator  use  the  tools  help  with  accuracy  and  speed. 

To  increase  the  effectiveness  of  the  tool  the  NASA  assessment  team  noted  that  linking  the  SEP  AT  and 
SECAT  to  specific  program  success  criteria  would  be  of  significant  value.  This  extension  to  the 
research  could  be  approached  in  two  ways.  The  first  would  be  to  link  the  tools  to  general  program 
success  criteria  such  as  technical  requirements,  budget  and  schedule.  These  would  be  generic  measures 
common  to  all  programs.  The  second  approach  would  be  to  link  the  tools  to  specific  technical  measures 
such  as  risk  metrics,  engineering  change  notices,  test  results,  and  requirements  stability.  This  second 
approach  would  be  a  significant  body  of  work,  but  would  ultimately  be  more  helpful  in  linking  the 
payback  of  investing  in  systems  engineering  to  specific  technical  perfonnance  measures  rather  than 
general  programmatic  results. 


E.  SADB-DAPS  Based  SEPAT  and  SECAT  Evaluation  Summary 

A  study  was  conducted  to  determine  the  relationship  of  SEPAT’s  system  engineering  effectiveness 
measures  with  the  existing  Department  of  Defense’s  Defense  Acquisition  Program  Support  (DAPS) 
Methodology  [22]  and  its  associated  Systemic  Analysis  Database  (SADB). 

In  general,  the  study  found  the  two  frameworks  to  be  complementary.  The  SEPAT  can  be  characterized 
of  as  a  subset  of  the  topics  covered  in  the  DAPS.  Initial  analysis  of  a  large  set  of  SADB  findings 
revealed  interesting  framework  comparison  details  and  indicates  an  opportunity  for  further  research. 
Supplementary  data  comparisons  and  analysis  of  additional,  relevant  SADB  findings  is  needed  to 
complete  the  analysis.  Study  results  indicate  that  the  two  frameworks  are  synergistic  and  may  be 
leveraged  as  complements  in  the  evaluation  of  MDAPS. 


1.  SEPAT  and  DAPS/SADB  Study  Objectives  and  Approach 

The  objectives  of  this  study  were  to: 
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•  Compare  SEP  AT  with  DAPS  to  determine  gaps  and  coverage. 

•  Establish  a  baseline  for  revisions  and  future  comparisons. 

•  Provide  constructive  feedback  to  both  model  owners  as  appropriate. 

•  Detennine  the  significance  of  SEP  AT  effectiveness  measures  as  they  relate  to  SADB  negative 
findings. 

•  Understand  whether  it  is  possible  to  validate  the  set  of  SEP  AT  systems  engineering  effectiveness 
measures  through  an  analysis  of  SADB  findings,  that  is  evaluate  the  ability  of  the  SEPAT  to  pre¬ 
identify  MDAP  problem  areas  articulated  in  the  SADB. 

Initial  research  focused  on  two  primary  activities:  (1)  mapping  DAPS  and  SEPAT  topical  areas  and  (2) 
analyzing  SEPAT-related  findings  in  the  SADB.  These  activities  were  perfonned  by  very  experienced 
subject  matter  experts  with  deep  system  development  and  federal  acquisition  experience;  however,  the 
tasks  were  labor  intensive  and  tedious  in  nature.  Consequently,  these  initial  results  are  not  exhaustive 
and  further  validation  and  verification  is  required. 

The  approach  used  to  map  the  DAPS  and  SEPAT  frameworks  included  identifying  and  selecting  the  key 
topical  areas,  searching  the  text  in  each  framework  using  key  words,  documenting  these  results  in  a 
Microsoft  Excel  spreadsheet,  and  analyzing  and  reviewing  the  resulting  comparison  map.  In  parallel,  an 
investigation  of  the  SADB  findings  was  perfonned  to  explore  the  use  of  the  SADB  in  evaluating  the 
ability  of  the  SEPAT  to  pre-identify  SADB  problem  areas.  This  included  submitting  a  request  for  SADB 
findings,  sorting  and  analyzing  the  data  set,  and  comparing  the  findings  with  the  SEPAT. 


2.  Existing  Defense  Acquisition  Program  Support  Tools 

The  Defense  Acquisition  Program  Support  (DAPS)  Methodology  was  developed  by  the  US  Department 
of  Defense,  Office  of  the  Deputy  Under  Secretary  of  Defense  for  Acquisition  and  Technology  (OUSD 
(AT&L)),  Systems  and  Software  Engineering.  The  most  recent  version,  2.0  (Change  3),  was  published 
March  20,  2009.  The  objectives  of  this  methodology  are  to: 

•  Improve  the  OUSD(AT&L)  decision-making  process  for  Major  Defense  Acquisition  Programs 
(MDAS)  and  Major  Automated  Infonnation  Systems  programs  through  quality  systems  engineering 
and  program  assessment  support. 

•  Facilitate  successful  execution  of  a  program  through  the  provision  of  independent,  actionable 
recommendations  to  the  government  program  management  office  (PMO). 

The  DAPS  Methodology  is  used  in  a  very  specific  Department  of  Defense  context.  In  this  context,  the 
methodology 

•  Provides  the  tailorable  framework  for  conducting  Program  Support  Reviews  (PSRs)  to  assist 
program  managers  and  DoD  decision  makers  in  preparation  for  milestone  decision  reviews. 

•  Provides  a  standardized  approach  (detailed  review  typology)  to  conducting  PSRs,  allowing  for  the 
participation  of  a  broad  cadre  of  subject  matter  experts  while  expecting  the  same  level  of  coverage 
and  quality  among  all  reviews. 
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•  Enabled  the  creation  of  the  Systemic  Analysis  Database  (SADB)  of  program  issues  and  root  causes. 
This  database  contains  findings  from  reviews  conducted  using  DAPS  and  allows  systemic  analyses 
that  can  be  used  to  effect  improvements  to  the  acquisition  process  (e.g.,  policies,  tools,  and 
education)  and  identify  best  practices. 

The  DAPS  Methodology  topical  area  content  focuses  on  systems  engineering,  but  covers  a  broader  range 
of  subjects  in  consideration  all  aspects  of  acquisition  management,  including  resource  planning, 
management  methods  and  tools,  earned  value  management,  logistics,  and  other  areas.  The  methodology 
is  composed  of  a  robust  listing  of  programmatic  and  technical  areas,  sub-areas,  and  factors.  It  was 
developed  to  be  both  broad  in  scope  and  detailed  enough  to  enable  application  to  programs  of  all  types. 

The  First-Level  Programmatic  and  Technical  Areas  are  defined  as  follows: 

1 .  Mission  Capabilities 

2.  Resources 

3 .  Management 

4.  Technical  Process 

5.  Performance 

6.  Special  Interest  Areas 

The  DAPS  Methodology  provides  a  complete  description  of  each  programmatic  and  technical  area  as 
well  as  its  intended  use  and  processes. 


3.  SEPAT  -  DAPS  Framework  Mapping  Results 

Raw  data  results  of  the  mapping  between  the  SEPAT  and  DAPS  frameworks  were  documented  in  a 
Microsoft  Excel  spreadsheet.  Figure  1  provides  a  sample  of  this  mapping.  A  complete  mapping  of  the 
SEPAT  and  DAPS  frameworks  can  be  found  in  an  attachment  to  this  report.  As  indicated  previously, 
these  initial  results  are  not  exhaustive  and  further  validation  and  verification  is  required.  However,  it  can 
be  detennined  that  the  two  frameworks  overlap  extensively  and  nearly  each  area  in  the  SEPAT  can  be 
traced  to  the  DAPS  in  some  fashion. 


33 


SE  EM  Framework  Area 

Interpretation 

DAPS 

Section 

DAPS  Topic  Covered 

Comments/Observations 

1  Concurrent  definition  of  system  requirements  and 
solutions 

1.1 

Understanding  of  stakeholder  needs:  capabilities, 
operational  concept,  key  performance  parameters, 
enterprise  fit  (legacy) 

1.1 

CONOPS 

1. 

1. 

1 

At  Milestone  A,  have  the  KPPs  been  identified  in  clear, 
comprehensive,  concise  terms  that  are  understandable 
to  the  users  of  the  system? 

Understandable, 

comprehensive 

requirements 

1.3.2 

KPPs  and  KSAs 

KPPs  are  required  to  be  "established  and 
documented";  no  guidance  on 
understandability/quality  of  KPP;  however 
states-> 

1. 

1. 

2 

Has  a  CONOPS  been  developed  showing  that  the 
system  can  be  operated  to  handle  both  nominal  and 
off-nominal  workloads,  and  to  meet  response  time 
requirements? 

feasible  workload 

demonstrated  at 

CONOPS;  are  scenarios 
quantified 

1.1 

1.2 

1.3.1 

CONOPS 

Analysis  of  Alternatives 

Reasonableness,  Stability,  and  Testability 

DAPS  specifies  "realistic"  scenarios,  not 
necessarily  "nominal  and  off-nominal 
workloads."  These  terms  and  "response 
time"  are  more  network-oriented. 
Quantification  is  given  by  "measures  of 
effectiveness." 

1. 

1. 

3 

Has  the  ability  of  the  system  to  meet  mission 
effectiveness  goals  been  verified  through  the  use  of 
modeling  and  simulation? 

verification  of  mission 
goals 

1.2.1 

1.3.1 

4.4.2 

Validity  and  Currency  (1.2  Analysis  of 
Alternatives) 

Reasonableness,  Stability,  and  Testability 
Modeling  and  Simulation  Tools 

Figure  10.  SEPAT  -  DAPS  Mapping  Sample 

Some  summary  observations  include: 

•  The  SEPAT  has  ‘rolled  up’  many  subtopics  into  fewer  effectiveness  areas;  DAPS  is  more  broad  and 
more  specific 

-  Specific  issues  (e.g.,  complementary  programs)  are  called  out  more  explicitly  in  DAPS 
and  handled  more  generically  as  ‘risks’  in  SEPAT. 

-  DAPS  discusses  DoD-specific  artifacts  and  products  (e.g.,  those  required  for  DoDAF, 
CARD,  JCIDS,  DIACAP). 

•  Several  key  concepts  found  in  SEPAT  seem  to  be  absent  or  used  in  a  different  manner  in  the  DAPS 
context,  such  as 

-  Negotiating  roles  &  responsibilities 

-  Nominal/off-nominal 

-  Stakeholder  concurrence/acceptance 

-  Validated 

-  Feasibility  evidence 

-  Timeboxing 

•  DAPS  is  very  DoD-process  oriented;  as  a  result,  its  guidance  on  what  the  program  should  do  when 
is  more  specific,  e.g.,  each  section  addresses  Milestones  A,  B,  and  C  as  appropriate. 

•  DAPS  has  more  information  about  expected  implementation,  often  providing  a  discussion  of  the 
rationale  or  importance  of  the  topic. 
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•  DAPS  makes  several  assumptions  regarding  the  programs’  required  characteristics,  processes,  such 
as  the  required  use  of  Modular  Open  Systems  Approach,  DoDAF,  and  Earned  Value  Management. 
These  types  of  assumptions  are  not  generally  found  in  SEP  AT. 

Detailed  differences  between  the  frameworks  are  described  below: 

•  SEP  AT  does  not  address  ‘ilities’  as  independent  topics  as  addressed  in  DAPS: 

-  5.2  Suitability 

■  5.2.1  Reliability  Assessment 

■  5.2.2  Availability  Assessment 

■  5.2.3  Maintainability  Assessment 

-  5.3  Survivability 

•  SEPAT  does  not  cover  domain  specifics  or  specialty  engineering  areas,  such  as  Security; 
Information  Assurance;  Weapons  Systems;  Spectrum  Management;  Human  Systems  Integration; 
Environment,  Safety,  Occupational  Health;  and  Corrosion. 

•  SEPAT  does  not  address  topics  beyond  the  development  phase,  including  production,  logistics, 
maintenance  upgrades. 

•  DAPS  often  mentions  an  incremental  approach,  but  it  does  not  address  how  to  divide  requirements 
into  increments  or  prioritize  requirements;  advocates  open  systems  (1.4,  1.4.4,  3.4.4). 

•  DAPS  requires  KPPs  be  established  and  documented,  but  there  is  no  guidance  on  the 
understandability/quality  of  KPP  (1.1.1). 

•  DAPS  does  not  explicitly  require  key  stakeholder  agreement/acceptance  on  the  system  boundary 
and  assumptions  of  its  environment  though  it  does  expect  that  collaboration  mechanisms  are  in 
place  (1.3.4,  2.3.1,  2.3.2,  2.4.1,  3.4.2,  4.5.4). 

•  DAPS  does  not  address  ‘negotiation’  of  roles  and  responsibilities,  but  rather  assumes  the  PM 
identifies  what  needs  to  be  done  and  who  shall  address  it  (1.1.4). 

•  DAPS  does  not  specifically  address  the  timeframe  of  personnel  assignments,  although  strategy  and 
schedule  realism  is  emphasized  (1.4.1,  2.1.3,  2.5.3). 

•  DAPS  does  address  quality  of  staff  in  general,  but  does  not  specifically  address  staff  needed  in 
critical  areas  or  what  constitutes  a  qualified  person  (2.1.1,  2.2.2). 

•  DAPS  has  an  Overarching  Integrated  Product  Team  but  it  is  unclear  on  this  group's  purpose  or 
participation  in  the  program  during  all  life  cycles;  a  "super  IPT“  is  not  mentioned,  but  DAPS  uses 
planning  and  reviews  to  resolve  issues  (2.2.3). 

•  DAPS  specifies  ‘realistic’  scenarios,  but  not  ‘nominal’  and  ‘off-nominal’  workloads  (1.1.2). 
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DAPS  is  not  specific  as  SEP  AT  regarding  network-oriented  performance,  e.g.,  ‘response  times’ 

(1.1.2). 


•  DAPS  considers  flexibility  relative  to  system  design,  but  does  not  focus  on  the  ability  of 
requirements  to  take  on  future  mission  growth  over  the  program  lifecycle  (1.4.2). 

•  DAPS  seems  to  recommend  Modular  Open  Systems  Approach  as  an  approach  to  resolve  conflicts 
among  strongly  coupled  objectives  (2.3.3). 

•  DAPS  does  not  address  COTS  validation,  though  it  does  address  COTS  suitability  through  an 
acceptance  process  (1.2.4). 

•  DAPS  mentions  prototypes  as  part  of  the  SDD  process,  but  does  not  explicitly  mention  their  use  for 
mitigating  risks  (1.4.3). 

•  DAPS  addresses  competitive  prototyping  and  expects  the  results  to  be  reviewed  at  major  reviews, 
but  it  is  unclear  how  to  plan  for  it  from  a  contracting,  PM  perspective  (2.4.1,  2.4.2). 

•  DAPS  does  not  address  effective  strategies  for  addressing  proposed  changes,  such  as  triage  (4.4.1). 

•  DAPS  relates  the  expected  level  of  formality  with  size  of  the  project  in  terms  of  cost  versus  stability 
of  requirements  (4.1.1). 

•  DAPS  does  not  appear  to  have  an  explicit  requirement  for  traceability  between  requirements  and 
architecture  (3.2.4). 

•  DAPS  does  not  address  the  concept  of  time-determined  development,  though  it  does  expect 
schedule  constraints  are  dealt  with  and  recommends  a  schedule  reserve  (3.4.4). 

•  DAPS  discusses  milestone  reviews  at  length;  however,  it  is  not  clear  that  evidence  of  feasibility  is 
available  and  ‘checking  the  box’  is  explicitly  avoided  (4.5). 

•  DAPS  does  not  explicitly  discuss  feasibility  evidence,  and  therefore  does  not  address  related 
progress  measures  (4.2.4). 

•  DAPS  discusses  milestone  reviews  at  length;  however,  it  is  not  clear  that  evidence  of  feasibility  is 
available  and  ‘checking  the  box’  is  explicitly  avoided  (4.5). 

It  should  be  noted  that  these  differences  are  not  described  as  positive  or  negative.  It  is  clear  that  the 
DAPS  Methodology  has  been  carefully  crafted  to  incorporate  the  constraints  and  complexities  of 
systems  development  within  the  DoD  context.  Discussions  with  the  DAPS  Methodology  owners  are 
required  to  determine  whether  there  is  opportunity  or  rationale  for  incorporating  SEP  AT  key  concepts, 
for  example,  into  the  DAPS  and  conversely,  SEPAT  use  of  DAPS  elaborations. 


4.  SADB  Analysis  Results 

A  request  for  SADB  findings  was  requested  in  accordance  with  the  SADB  Report  Request  Form.  The 
data  requested  was  intended  to  be  related  to  SEPAT  Area  1:  Concurrent  Definition  of  System 
Requirements  and  Solutions.  The  following  DAPS  Sections  were  indicated  on  the  request  form: 
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1  Mission  Capabilities  (1.1  CONOPS,  1.2  Analysis  of  Alternatives,  1.3  Capabilities) 

2.2  Budget  Sufficiency  and  Phasing 

3 . 1  Acquisition  Strategy 

3.2  Knowledge-Based  Decisions  and  Milestones 

4.5.2  Verific  ation  C  orrelation 

4.6  Design  Verification 

4.7  Supportability  Planning 

This  data  request  represents  9  of  59  DAPS  factors  for  comparison.  In  addition  to  the  checked  areas,  the 
request  included  the  following  additional  key  words:  Increments;  Understandable,  comprehensive, 
concise  requirements;  Verification  of  mission  goals;  Stakeholder  roles;  Legacy,  context,  operational 
concept  verification;  Exploration  of  alternative  solutions;  External  interfaces;  Third-party  solutions; 
Prioritization  of  requirements;  System  development. 

A  total  of  1,412  findings  were  received  in  response  to  this  request.  Of  these  findings,  704  were  classified 
as  Negative  Findings,  491  were  Neutral  Findings,  and  217  were  Positive  Findings.  Figure  11  illustrates 
the  characterization  of  the  data  set. 


Figure  11.  Data  Set 

Additional  data  characterization  observations  regarding  the  data  received  include: 

•  The  findings  received  represent  programs  sponsored  by  the  Army,  Navy,  Air  Force,  Marine  Corps, 
as  well  as  other  DoD  components. 

•  The  largest  group  of  findings  is  related  to  DAPS  section  1.3.1:  Reasonableness,  Stability, 
Testability. 

•  Of  AFF  findings  received,  nearly  half  or  more  are  related  to  each  DAPS  section  are  Negative 
Findings,  except  those  related  to  the  following  areas: 


Mission  Description 

1.2.1  Validity  and  Concurrency 
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-  1.2.2  Linkage  and  Traceability 

-  4.7.2  Performance-Based  Logistics 
•  About  74%  of  the  NEGATIVE  findings 

-  Have  50  or  more  findings  related  to  its  DAPS  section 

-  Cover  the  following  seven  (7)  DAPS  areas: 

■  1.3.1  Reasonableness,  Stability,  Testability 

■  2.2.1  Program  Funding  and  Allocation 

■  Credibility 

■  Acceptability 

■  4.6. 1  Test  and  Evaluation  Plan 

■  4.6.2  Verification  Correlation 

■  4.7.3  Sustainment 


5.  Recommendations  and  Conclusions 

The  initial  results  of  the  study  indicate  that  the  DAPS  and  SEP  AT  frameworks  are  complementary. 
There  is  a  great  deal  of  overlap  between  the  frameworks  and  there  are  opportunities  to  leverage  details 
in  each  framework.  The  large  number  of  relevant  SADB  findings  (over  1,400)  indicates  synergy 
between  the  frameworks.  Specific  considerations  for  improvements  to  SEPAT  include: 

•  Clarify  the  program  life  cycle  assumed  by  the  SEPAT  framework,  for  example,  milestone  events 
and  timeline. 

•  Include  definitions  with  the  SEPAT  for  terms  not  currently  used,  e.g.,  mission  effectiveness; 
concurrent  solution;  feasibility  evidence;  quality  of  service;  program  governance  process;  level  of 
project  requirements  emergence;  earned  value  target. 

•  More  specific  guidance  is  needed  regarding  what  is  meant  by  ‘ validated ’  requirements,  quality  of 
service  guarantees,  and  solutions,  and  ‘ clearly  demonstrated  compliance ’  with  legal,  policy, 
regulatory,  standards,  security,  etc. 

•  Clarify  how  to  interpret  the  SEPAT  given  a  specific  perspective,  e.g.,  government  PMO  staffer 
versus  system  development  contractor. 

•  Provide  discussion  on  key  concepts,  especially  those  that  are  under-addressed  by  DAPS. 

•  Include  guidance  or  references  to  assist  in  understanding  and  implementation  of  the  SEPAT. 
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•  Differentiate  the  issues  cited  in  2.4.3:  resources,  process  infrastructure,  organizational  viability, 
project  alignment  with  business  unit. 

•  Separate  2.5.2  issues  related  to  empowerment  and  qualified  persons. 

•  Reword  3.2. 1  to  be  more  system  oriented  versus  software. 

•  Clarify  meaning  of  3.2.5:  “Does  the  architecture  adequately  reconcile  functional  hardware  “part-of  ’ 
hierarchies  with  layered  software  “served-by”  hierarchies?” 

•  Change  earned  value  targets  to  earned  value  thresholds  in  4.2.4. 

As  mentioned  earlier,  it  is  recommended  that  discussions  with  DAPS  Methodology  owners  be  conducted 
to  provide  the  details  of  this  study  to  determine  if  there  are  any  salient  improvements  that  may  be  made 
to  the  DAPS. 

Additionally,  it  is  recommended  that  further  analysis  of  SADB  findings  be  conducted  to  leverage  the 
existing  data  and  refine  the  set  of  SEP  AT  effectiveness  measures. 


F.  Conclusions  and  Recommendations 

1.  Conclusions 

The  results  of  the  SEP  AT  and  SECAT  pilot  assessments,  the  DAPS  and  SADB  comparative  analysis, 
and  the  quantitative  business  case  analysis  for  the  use  of  the  SE  EM  framework,  tools,  and  operational 
concepts  is  sufficiently  positive  to  conclude  that  implementation  of  the  approach  is  worth  pursuing. 
Presentations  at  recent  workshops  have  generated  considerable  interest  in  refining,  using,  and  extending 
the  capabilities  and  in  co-funding  the  followon  research.  However,  the  framework  and  prototype  tools 
have  been  shown  to  be  largely  efficacious  only  to  date  for  pilot  projects  done  by  familiar  experts  in  a 
relatively  short  time.  It  remains  to  demonstrate  how  well  the  framework  and  tools  will  perform  on  in- 
process  MDAPs  with  multiple  missions,  perfonners,  and  independent  expert  assessors. 

The  parametric  analysis  in  Section  A. 3  concludes  that  the  greater  the  project’s  size,  criticality,  and 
stability  are,  the  greater  is  the  need  for  validated  architecture  feasibility  evidence  (i.e.,  evidence-based 
specifications  and  plans).  However,  for  very  small,  low-criticality  projects  with  high  volatility,  the 
evidence  generation  efforts  would  make  little  difference  and  would  need  to  be  continuously  redone, 
producing  a  negative  return  on  investment.  In  such  cases,  agile  methods  such  as  rapid  prototyping, 
Scrum  and  extreme  Prograimning  will  be  more  effective.  Overall,  evidence-based  specifications  and 
plans  will  not  guarantee  a  successful  project,  but  in  general  will  eliminate  many  of  the  software  delivery 
overruns  and  shortfalls  experienced  on  current  software  projects. 

Some  implications  of  defining  feasibility  evidence  as  a  “first  class”  project  deliverable  are  that  it  needs 
to  be  planned  (with  resources),  and  made  part  of  the  project’s  earned  value  management  system.  Any 
shortfalls  in  evidence  are  sources  of  uncertainty  and  risk,  and  should  be  covered  by  risk  management 
plans.  The  main  contributions  of  the  SERC  SE  EM  project  have  been  to  provide  experience-based 
approaches  and  operational  concepts  for  the  use  of  evidence  criteria,  evidence-generation  procedures, 
and  SE  effectiveness  measures  for  monitoring  evidence  generation,  which  support  the  ability  to  perform 
evidence-based  SE  on  DoD  MDAPs.  And  finally,  evidence-based  specifications  and  plans  such  as  those 
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provided  by  the  SERC  SE  EM  capabilities  and  the  Feasibility  Evidence  Description  can  and  should  be 
added  to  traditional  milestone  reviews. 

As  a  bottom  line,  the  SERC  SE  capabilities  have  strong  potential  for  transforming  the  largely 
unmeasured  DoD  SE  activity  content  on  current  MDAPs  and  other  projects  into  an  evidence-based 
measurement  and  management  approach  for  both  improving  the  outcomes  of  current  projects,  and  for 
developing  a  knowledge  base  that  can  serve  as  a  basis  for  continuing  DoD  SE  effectiveness 
improvement. 


2.  Recommendations 

Based  on  the  Conclusions,  we  recommend  a  two-step  approach  for  achieving  a  SE  EM  initial 
operational  capability  and  transitioning  it  to  a  sustaining  organization.  Phase  Ha  is  proposed  to  begin 
with  research  on  three  tasks.  The  first  task  would  involve  experimentation  with  domain  extensions  and 
larger-scale  pilots.  The  second  task  would  involve  performing  and  analyzing  the  results  of  further 
completed  successful  and  unsuccessful  projects  to  test  the  hypothesis  that  there  is  a  critical  small  set  of 
critical  success-failure  factors  that  could  serve  as  top-level  early  warning  indicators.  The  third  task 
would  involve  extended  analyses  of  the  commonalities  and  variabilities  between  the  SERC  SE  EM 
apabilities  and  the  DAPS  methodology  and  SADB  results.  This  could  strengthen  both  and  enable  them 
to  be  used  in  complementary  ways. 

Phase  lib  would  continue  with  incremental  elaboration,  experimentation,  and  refinement  of  the  preferred 
approaches,  and  coordination  with  complementary  efforts.  Candidate  tasks  would  include  EM  tool  top- 
risk  summaries,  suggestions  for  mitigating  the  identified  risks,  ease  of  creating  domain-specific 
extensions,  creating  further  users-guide  and  tutorial  material,  creating  and  populating  a  knowledge  base 
of  the  results,  plans  for  transitioning  the  support  and  evolution  of  the  tools  to  an  appropriate  support 
organization  such  as  DAU,  and  continuing  to  coordinate  the  tools’  content  with  complementary 
initiatives  such  as  the  INCOSE  Leading  Indicators  upgrade,  the  NDIA  enterprise-oriented  personnel 
competency  initiative,  and  the  SERC  SE  Body  of  Knowledge  and  Reference  Curriculum  RT. 
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MIT  -  Massachusetts  Institute  of  Technology 

MPT  -  Methods,  Processes,  and  Tools 

NDI  -  Non-Development  Item 

NDIA  -  National  Defense  Industrial  Association 
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NRC  -  National  Research  Council 


NUWC  -  Naval  Undersea  Warfare  Center 

OUSD(AT&L)  -  Office  of  the  Under  Secretary  of  Defense  (Acquisition,  Technology,  and  Logistics) 

PEO-PM  -  Program  Executive  Officer  -  Program  Manager 

PDR  -  Preliminary  Design  Review 

PoPS  -  Probability  of  Program  Success 

PSTL  -  Program  Support  Team  Leader 

RDT&E  -  Research,  Development,  Testing  &  Evaluation 

RAA  -  Responsibility,  Authority,  and  Accountability 

RESL  -  COCOMO  II  Architecture  and  Risk  Resolution  Factor 

ROI  -  Return  on  Investment 

RUP  -  Rational  Unified  Process 

SADB  -  Systemic  Analysis  Database 

SE  -  Systems  Engineering 

SEC  AT  -  Systems  Engineering  Competency  Assessment  Tool 

SEI-CMMI  -  Software  Engineering  Institute  Capability  Maturity  Model  Integration 

SEP  AT  -  Systems  Engineering  Performance  Assessment  Tool 

SERC  -  Systems  Engineering  Research  Center 

SISAIG  -  Software  Intensive  Systems  Acquisition  Improvement  Group 

UAH  -  University  of  Alabama  in  Huntsville 

UARC  -  University  Affiliated  Research  Center 

UML  -  Unified  Modeling  Language 

USC  -  University  of  Southern  California 
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APPENDIX  A:  GOALS,  CRITICAL  SUCCESS  FACTORS,  AND 

QUESTIONS 

This  section  has  the  Goals,  Critical  Success  Factors,  and  Questions  for  both  the  SEP  AT  and  the  SECAT. 


SEP  AT  -  Goals,  Critical  Success  Factors,  and  Questions 


Goal  1.  Concurrent  definition  of  system  requirements  and  solutions 

CSF  1.1  Understanding  of  stakeholder  needs:  capabilities,  operational  concept,  key 
performance  parameters,  enterprise  fit  (legacy) 

(a)  At  Milestone  A,  have  the  Key  Performance  Parameters  (KPPs)  been  identified  in  clear, 
comprehensive,  concise  terms  that  are  understandable  to  the  users  of  the  system? 

(b)  Has  a  Concept  of  Operations  (CONOPS)  been  developed  showing  that  the  system  can  be 
operated  to  handle  both  nominal  and  off-nominal  workloads  and  meet  response  time 
requirements? 

(c)  Has  the  ability  of  the  system  to  meet  mission  effectiveness  goals  been  verified  through  the  use 
of  modeling  and  simulation? 

(d)  Have  the  success-critical  stakeholders  been  identified  and  their  roles  and  responsibilities 
negotiated? 

(e)  Have  questions  about  the  fit  of  the  system  into  the  stakeholders'  context — acquirers,  end  users, 
administrators,  interoperators,  maintainers,  etc. — been  adequately  explored? 


CSF  1.2  Concurrent  exploration  of  solution  opportunities;  Analysis  of  Alternatives  (AoAs) 
for  cost-effectiveness  and  risk  (measures  of  effectiveness) 

(a)  Have  at  least  two  alternative  approaches  been  explored  and  evaluated? 

(b)  At  Milestone  B,  has  the  government  structured  the  program  plan  to  ensure  that  the  contractor 
addresses  the  allocation  of  capabilities  to  hardware,  software,  and  human  elements  sufficiently 
early  in  the  development  program? 

(c)  Has  the  claimed  degree  of  reuse  been  validated? 

(d)  Have  the  claimed  quality  of  service  guarantees  been  validated? 
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(e)  Have  proposed  Commercial  Off-the-Shelf  (COTS)  and  third-party  solutions  been  validated  for 
maturity,  compatibility,  supportability,  suitability,  and  effectiveness,  throughout  the  expected 
system  lifetime? 


CSF  1.3  System  scoping  &  requirements  definition  (external  interfaces;  Memoranda  of 
Agreement  (MoA)) 

(a)  Have  external  interface  complexities  been  identified  and  addressed  via  MoAs  or  their 
equivalent?  Is  there  a  plan  to  mitigate  their  risks? 

(b)  At  Milestone  B,  are  the  major  system-level  requirements  (including  all  KPPs)  defined 
sufficiently  to  provide  a  stable  basis  for  the  development  through  Initial  Operational  Capability 
(IOC)? 

(c)  By  Milestone  A,  is  there  a  plan  to  have  infonnation  exchange  protocols  established  for  the  whole 
system  and  its  segments  by  Milestone  B? 

(d)  Have  the  key  stakeholders  agreed  on  the  system  boundary  and  assumptions  about  its 
environment? 


CSF  1.4  Prioritization  of  requirements  &  allocation  to  increments 

(a)  Can  an  initial  capability  be  achieved  within  the  time  that  the  key  program  leaders  are  expected  to 
remain  engaged  in  their  current  jobs  (normally  less  than  5  years  or  so  after  Milestone  B)?  If  this 
is  not  possible  for  a  complex  major  development  program,  can  critical  subsystems,  or  at  least  a 
key  subset  of  them,  be  demonstrated  within  that  time  frame? 

(b)  At  Milestone  B,  do  the  requirements  and  proposed  solutions  take  into  account  likely  future 
mission  growth  over  the  program  life  cycle? 

(c)  Have  appropriate  early  evaluation  phases,  such  as  competitive  prototyping,  been  considered  or 
executed  for  high-risk/low-maturity  components  of  the  system? 

(d)  Have  stakeholders  agreed  on  prioritization  of  system  features  and  their  allocation  to  development 
increments? 


Goal  2.  System  life-cycle  organization,  planning,  and  staffing 

CSF  2.1  Establishment  of  stakeholder  life-cycle  Responsibilities,  Authorities,  and 
Accountabilities  (RAAs)  (for  system  definition  &  system  development) 
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(a)  Are  the  stakeholders  who  have  been  identified  as  critical  to  the  success  of  the  project  represented 
by  highly  qualified  personnel  —  those  who  are  collaborative,  representative,  empowered, 
committed,  and  knowledgeable? 

(b)  At  Milestone  A,  are  there  validated  plans,  budgets,  and  schedules  defining  how  the  pre- 
Milestone  B  activity  will  be  done,  and  by  whom? 

(c)  Has  the  government  attempted  to  align  the  duration  of  the  program  manager's  assignment  with 
key  deliverables  and  milestones  in  the  program? 

(d)  Have  the  key  stakeholders  agreed  to  the  proposed  assignments  of  system  roles,  responsibilities, 
and  authorities? 


CSF  2.2  Establishment  of  Integrated  Product  Team  (IPT)  RAAs,  cross-IPT  coordination 
needs 

(a)  Does  the  project  make  effective  use  of  Integrated  Project  Teams  (IPTs)  throughout  the  supplier 
hierarchy? 

(b)  Are  the  IPTs  staffed  by  highly  qualified  personnel,  as  in  2. 1  (a)? 

(c)  For  IPTs  addressing  strongly  coupled  objectives,  are  there  super-IPTs  for  resolving  conflicts 
among  the  objectives? 


CSF  2.3  Establishment  of  necessary  plans  and  resources  for  meeting  objectives 

(a)  Have  decisions  about  the  use  of  one-shot,  incremental,  or  evolutionary  development  been 
validated  for  appropriateness  and  feasibility,  and  accepted  by  the  key  stakeholders? 

(b)  Have  system  definition,  development,  test,  and  evolution  plans,  budgets,  and  schedules  been 
validated  for  appropriateness  and  feasibility,  and  accepted  by  the  key  stakeholders? 

(c)  Is  there  a  valid  business  case  for  the  system,  relating  the  life  cycle  system  benefits  to  the  system 
total  cost  of  ownership? 


CSF  2.4  Establishment  of  appropriate  source  selection,  contracting,  and  incentive  structures 

(a)  Has  the  competitive  prototyping  option  been  addressed,  and  the  decision  accepted  by  the  key 
stakeholders? 

(b)  If  doing  competitive  prototyping,  have  adequate  plans  and  preparations  been  made  for  exercising 
and  evaluating  the  prototypes,  and  for  sustaining  core  competitive  teams  during  evaluation  and 
downselecting? 
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(c)  Is  the  status  of  the  candidate  perfonner's  business  and  team  "healthy,"  both  in  terms  of  business 
indicators,  and  within  the  industrial  base  for  the  program  area?  Is  the  program  aligned  with  the 
core  business  of  the  unit,  and  staffed  adequately  and  appropriately? 

(d)  Has  the  acquiring  organization  successfully  completed  projects  similar  to  this  one  in  the  past? 

(e)  Has  the  candidate  perfonning  organization  successfully  completed  projects  similar  to  this  one  in 
the  past? 

(f)  Is  the  program  governance  process,  and  in  particular  the  system  engineering  plan,  well 
articulated  and  compatible  with  the  goals  of  the  program? 


CSF  2.5  Establishment  of  necessary  personnel  competencies 

(a)  Does  the  government  have  access  over  the  life  of  the  program  to  the  talent  required  to  manage 
the  program?  Does  it  have  a  strategy  over  the  life  of  the  program  for  using  the  best  people 
available  in  the  government,  the  Federally  Funded  Research  and  Development  Centers 
(FFRDCs),  and  the  professional  service  industry? 

(b)  At  Milestone  B,  have  sufficiently  talented  and  experienced  program  and  systems  engineering 
managers  been  identified?  Have  they  been  empowered  to  tailor  processes  and  to  enforce 
development  stability  from  Milestone  B  through  IOC? 

(c)  Has  the  government  attempted  to  align  the  duration  of  the  program  manager's  assignment  with 
key  deliverables  and  milestones  in  the  program? 

(d)  Is  the  quantity  of  developer  systems  engineering  personnel  assigned,  their  skill  and  seniority 
mix,  and  the  time  phasing  of  their  application  throughout  the  program  lifecycle,  appropriate? 


Goal  3.  Technology  Maturing,  Architecting 

CSF  3.1  COTS/Non-Development  Item  (NDI)/Services  evaluation,  selection,  validation  for 
maturity  &  compatibility 

(a)  Have  COTS/NDI/Services  opportunities  been  evaluated  prior  to  baselining  requirements? 

(b)  Have  COTS/NDI/Services  scalability,  compatibility,  quality  of  service,  and  life  cycle  support 
risks  been  thoroughly  addressed? 

(c)  Has  a  COTS/NDI/Services  life  cycle  refresh  strategy  been  developed  and  validated? 


CSF  3.2  Life-cycle  architecture  definition  &  validation 
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(a)  Has  the  system  been  partitioned  to  define  segments  that  can  be  independently  developed  and 
tested  to  the  greatest  degree  possible? 

(b)  By  Milestone  A,  is  there  a  plan  to  have  internal  and  external  information  exchange  protocols 
established  and  validated  for  the  whole  system  and  its  segments  by  Milestone  B? 

(c)  Does  the  project  have  adequate  processes  in  place  to  define  the  verification,  test  &  validation, 
and  acceptance  of  systems  and  system  elements  at  all  phases  of  definition  and  development? 

(d)  Is  there  a  clear,  consistent,  and  traceable  relationship  between  system  requirements  and 
architectural  elements?  Have  potential  off-nominal  architecture-breakers  been  addressed? 

(e)  Does  the  architecture  adequately  reconcile  functional  hardware  part-of  hierarchies  with  layered 
software  served-by  hierarchies? 

(f)  Has  a  Work  Breakdown  Structure  (WBS)  been  developed  with  the  active  participation  of  all 
relevant  stakeholders,  which  accurately  reflects  both  the  hardware  and  the  software  product 
structure? 


CSF  3.3  Use  of  prototypes,  exercises,  models,  and  simulations  to  determine  technological 
solution  maturity 

(a)  Will  risky  new  technology  mature  before  Milestone  B?  Is  there  a  risk  mitigation  plan? 

(b)  Have  the  key  non-technical  risk  drivers  been  identified  and  covered  by  risk  mitigation  plans? 

(c)  Is  there  a  sufficient  collection  of  models  and  appropriate  simulation  and  exercise  environments 
to  validate  the  selected  concept  and  the  CONOPS  against  the  KPPs? 

(d)  Has  the  claimed  degree  of  reuse  been  validated? 


CSF  3.4  Validated  system  engineering,  development,  manufacturing,  operations  & 
maintenance  budgets  and  schedules 

(a)  Are  the  major  known  cost  and  schedule  drivers  and  risks  explicitly  identified,  and  is  there  a  plan 
to  track  and  reduce  uncertainty? 

(b)  Have  the  cost  confidence  levels  been  developed  and  accepted  by  the  key  system  stakeholders? 

(c)  Is  there  a  top-to-bottom  plan  for  how  the  total  system  will  be  integrated  and  tested?  Does  it 
adequately  consider  integration  facilities  development  and  earlier  integration  testing? 

(d)  If  timeboxing  or  time-determined  development  is  used  to  stabilize  schedules,  have  features  been 
prioritized  and  the  system  architected  for  ease  of  adding  or  dropping  borderline  features? 
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(e)  Are  there  strategies  and  plans  for  evolving  the  architecture  while  stabilizing  development  and 
providing  continuity  of  service? 


Goal  4.  Evidence-based  progress  monitoring  and  commitment  reviews 

CSF  4.1  Monitoring  of  system  definition,  development  and  test  progress  vs.  plans 

(a)  Are  the  levels  and  fonnality  of  plans,  metrics,  evaluation  criteria,  and  associated  mechanisms 
(IMP,  IMS,  WBS,  EVMS)  commensurate  with  the  level  of  project  requirements  emergence  and 
stability?  (too  little  is  risky  for  pre-specifiable  and  stable  requirements;  too  much  is  risky  for 
emergent  and  unstable  requirements) 

(b)  Are  the  project's  staffing  plans  and  buildup  for  progress  monitoring  adequate  with  respect  to 
required  levels  of  expertise? 

(c)  Have  most  of  the  planned  project  personnel  billets  been  filled  with  staff  possessing  at  least  the 
required  qualification  level? 

(d)  Is  the  project  adequately  identifying  and  managing  its  risks? 

(e)  Have  the  processes  for  conducting  reviews  been  evaluated  for  feasibility,  reasonableness, 
completeness,  and  assurance  of  independence? 

(f)  Has  compliance  with  legal,  policy,  regulatory,  standards,  and  security  requirements  been  clearly 
demonstrated? 


CSF  4.2  Monitoring  of  feasibility  evidence  development  progress  vs.  plans 

(a)  Has  the  project  identified  the  highest  risk  areas  on  which  to  focus  feasibility  analysis? 

(b)  Has  the  project  analyzed  alternative  methods  of  evaluating  feasibility  (models,  simulations, 
benchmarks,  prototypes,  reference  checking,  past  performance,  etc.)  and  prepared  the 
infrastructure  for  using  the  most  cost-effective  choices? 

(c)  Has  the  project  identified  a  full  set  of  representative  operational  scenarios  across  which  to 
evaluate  feasibility? 

(d)  Has  the  project  prepared  milestone  plans  and  earned  value  targets  for  measuring  progress  in 
developing  feasibility  evidence? 

(e)  Is  the  project  successfully  monitoring  progress  and  applying  corrective  action  where  necessary? 
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CSF  4.3  Monitoring,  assessment,  and  replanning  for  changes  in  needs,  opportunities,  and 
resources 

(a)  Does  the  project  have  an  effective  strategy  for  performing  triage  (accept,  defer,  reject)  on 
proposed  changes,  that  does  not  destabilize  ongoing  development? 

(b)  Does  the  project  have  an  adequate  capability  for  performing  change  impact  analysis  and 
involving  appropriate  stakeholders  in  addressing  and  prioritizing  changes? 

(c)  Is  the  project  adequately  verifying  and  validating  proposed  changes  for  feasibility  and  cost- 
effectiveness? 


CSF  4.3  Use  of  milestone  reviews  to  ensure  stakeholder  commitment  to  proceed? 

(a)  Are  milestone  review  dates  based  on  availability  of  feasibility  evidence  versus  on  availability  of 
artifacts  or  on  planned  review  dates? 

(b)  Are  artifacts  and  evidence  of  feasibility  evaluated  and  risky  shortfalls  identified  by  key 
stakeholders  and  independent  experts  prior  to  review  events? 

(c)  Are  developer  responses  to  identified  risks  prepared  prior  to  review  events? 

(d)  Do  reviews  achieve  risk-based  concurrence  of  key  stakeholders  on  whether  to  proceed  into  the 
next  phase?  (proceed;  skip  a  phase;  revisit  the  current  phase;  tenninate  or  rescope  the  project) 
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SEC  AT  -  Goals,  Critical  Success  Factors,  and  Questions 


Goal  1.  Concurrent  definition  of  system  requirements  and  solutions 

CSF  1.1  Understanding  of  stakeholder  needs:  capabilities,  operational  concept,  key 

performance  parameters,  enterprise  fit  (legacy).  Ability  to  analyze  strengths  and 
shortfalls  in  current-system  operations  via: 

(a)  Participatory  workshops,  surveys,  focus  groups 

(b)  Operations  research  techniques:  operations  data  collection  and  analysis 

(c)  Mission  effectiveness  modeling  and  simulation 

(d)  Prototypes,  scenarios,  stories,  personas 

(e)  Ethnographic  techniques:  Interviews,  sampled  observations,  cognitive  task  analysis 


CSF  1.2  Concurrent  exploration  of  solution  opportunities;  Analysis  of  Alternatives  for  cost- 
effectiveness  &  risk  (Measures  of  Effectiveness).  Ability  to  identify  and  assess 
alternative  solution  opportunities  via  experimentation  and  analysis  of: 

(a)  Alternative  work  procedures,  non-materiel  solutions 

(b)  Purchased  or  furnished  products  and  services 

(c)  Emerging  technology 

(d)  Competitive  prototyping 


CSF  1.3  System  scoping  &  requirements  definition  (External  interfaces;  Memoranda  of 
Agreement).  Ability  to  establish  system  scope  and  requirements  via: 

(a)  Cost-schedule-effectiveness  assessment  of  needs  vs.  opportunities 

(b)  Organizational  responsibilities,  authorities,  and  accountabilities  (RAAs)  assessment 

(c)  Appropriate  degrees  of  requirements  completeness,  consistency,  testability,  and  variability  due 
to  emergence  considerations 
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CSF  1.4 


Prioritization  of  requirements  and  scheduling  into  increments.  Ability  to  prioritize 
requirements  and  schedule  them  into  increments  based  on  considerations  of: 


(a)  Stakeholder  priorities  and  returns  on  investment 

(b)  Capability  interdependencies  and  requirements  emergence  considerations 

(c)  Technology  maturity  and  implementation  feasibility  risks 

Goal  2.  System  Life  Cycle  Organization,  Planning,  Staffing 

CSF  2.1  Establishment  of  stakeholder  life  cycle  RAAs  for  system  definition,  system 

development,  and  system  operation.  Ability  to  support  establishment  of  stakeholder 
RAAs  via  conduct  of: 

(a)  Organizational  capability  analyses 

(b)  Stakeholder  negotiations 

(c)  Operational  exercise  analyses 


CSF  2.2  Establishment  of  Integrated  Product  Team  (IPT)  RAAs,  Cross-IPT  coordination 
needs.  Ability  to  establish  IPT  RAAs  and  cross-IPT  coordination  mechanisms  via: 

(a)  Risk  identification,  analysis,  and  prioritization 

(b)  Organizational  RAAs  and  skills  availability  assessment 

(c)  Risk  interdependency  analysis 

(d)  Risk  resolution  cost-benefit  analysis 


CSF  2.3  Establishment  of  necessary  resources  for  meeting  objectives.  Ability  to  support 
program  negotiation  of  objectives  vs.  resources  via: 

(a)  Cost-schedule-capability  tradeoff  analyses 

(b)  Use  of  requirements  priorities  and  interdependencies  to  support  negotiation  of  increment 
contents 

(c)  Development  of  strategies  to  adjust  increment  content  to  meet  delivery  schedules 

(d)  Analysis  of  project  change  traffic  and  rebaselining  of  future-increment  plans  and  specifications 
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CSF  2.4 


Establishment  and  usage  support  of  appropriate  source  selection,  contracting,  & 
incentive  structures.  Ability  to  support  program  management  in  preparing  source 
selection  materials,  matching  contracting  and  incentive  structures  to  program 
objectives,  and  technical  monitoring  of  performance  vs.  program  objectives: 

(a)  Preparation  of  proposal  solicitation  materials  and  evaluation  capabilities  and  procedures 

(b)  Evaluation  of  proposal  submissions  with  respect  to  criteria 

(c)  Technical  support  of  contract  negotiations 

(d)  Technical  support  of  contract  performance  monitoring 

CSF  2.5  Assurance  of  necessary  personnel  competencies.  Ability  to  support  program 
management  in  evaluating  proposed  staffing  plans  and  monitoring  staffing 
capabilities  vs.  plans  in  the  areas  of: 

(e)  Concurrent  Definition  of  System  Requirements  &  Solutions 

(f)  System  Life  Cycle  Organization,  Planning,  Staffing 

(g)  Technology  Maturing  and  Architecting 

(h)  Evidence-Based  Progress  Monitoring  &  Commitment  Reviews 

(i)  Professional  and  Interpersonal  Skills 


Goal  3.  Technology  Maturing  and  Architecting 

CSF  3.1  COTS/NDI/Services  evaluation,  selection,  validation  for  capability,  maturity  & 
compatibility.  Ability  to  evaluate  alternative  combinations  of  COTS,  NDI,  and 
purchased  services  for: 

(a)  Functional  capabilities  vs.  system  needs 

(b)  Levels  of  service:  perfonnance,  resilience,  scalability,  usability,  tailorability,  etc. 

(c)  Mutual  compatibility  and  external  interoperability 

(d)  Supplier  maturity,  stability,  support,  and  responsiveness 

(e)  Acquisition  and  operational  costs 


54 


CSF  3.2 


Life  Cycle  architecture  definition  &  validation.  Ability  to  define  and  evolve 
configurations  of  hardware  and  software  components  and  connectors  along  with 
human  operational  architectures,  and  to  validate  that  they  cost-effectively  support 
program  objectives: 

(a)  Define  candidate  hardware/software/human-operational  architectures 

(b)  Evaluate  their  functional  capabilities,  levels  of  service,  interoperability,  and  sustainability  vs. 
system  needs 

(c)  Perform  tradeoff  analyses  among  functional  capabilities  and  levels  of  service 


CSF  3.3  Use  of  prototypes,  exercises,  models,  and  simulations  to  determine  technology 

maturity,  architecture  feasibility.  Ability  to  assess  the  relative  costs  and  benefits  of 
alternative  evaluation  methods,  and  apply  appropriate  combinations  of  methods: 

(a)  Assess  relative  costs,  schedules,  and  capabilities  of  various  combinations  of  evaluation  methods 

(b)  Prepare  plans  for  enabling  and  performing  evaluations 

(c)  Prepare  representative  nominal  and  off-nominal  scenarios,  workload  generators,  virtual 
component  surrogates,  and  testbeds  to  support  evaluations 

(d)  Perform  evaluations,  analyze  results,  investigate  anomalies,  and  adjust  plans  as  appropriate 


CSF  3.4  Validated  System  Engineering,  Development,  Manufacturing,  Operations  & 
Maintenance  budgets  &  schedules.  Ability  to: 

(a)  Assess  alternative  budget  and  schedule  estimation  methods  vs.  nature  of  system,  degree  of 
system  knowledge,  complementarity  of  estimates,  and  cost  vs.  accuracy  of  performing  estimates 

(b)  Prepare  plans  for  gathering  inputs  and  performing  estimates 

(c)  Perform  selected  combinations  of  estimates  and  reconcile  their  differences 

(d)  Perform  tradeoff  analyses  among  functional  capabilities,  levels  of  service,  costs,  and  schedules 


Goal  4.  Evidence-Based  Progress  Monitoring  &  Commitment  Reviews 

CSF  4.1  Monitoring  of  system  definition,  development,  and  test  progress  vs.  plans.  Ability  to 
plan,  monitor,  and  evaluate  technical  progress  vs.  plans 
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(a)  Prepare  test  and  evaluation  facilities  and  plans,  and  define  data  to  be  provided  for  assessing 
technical  progress  vs.  project  plans 

(b)  Monitor  performers’  technical  progress  in  developing,  verifying,  and  validating  their  technical 
solutions 

(c)  Identify  shortfalls  in  technical  progress  vs.  plans,  and  determine  their  root  causes 


CSF  4.2  Monitoring  of  feasibility  evidence  development  and  test  progress  vs.  plans.  Ability 
to: 

(a)  Evaluate  developers’  feasibility  evidence  assessment  and  test  plans  for  coverage,  cost- 
effectiveness,  and  realism  of  assumptions 

(b)  Monitor  developers’  progress  with  respect  to  plans,  identify  shortfalls  and  root  causes 

(c)  Evaluate  feasibility  evidence  produced,  identify  shortfalls  and  root  causes 


CSF  4.3  Monitoring,  assessment,  and  replanning  for  changes  in  needs,  opportunities,  and 
resources.  Ability  to: 

(a)  Assess  proposed  changes  in  program  objectives,  constraints,  plans,  and  resources 

(b)  Perform  triage  to  determine  which  changes  should  be  handled  immediately,  deferred  to  future 
increments,  or  rejected 

(c)  Perform  tradeoff  analyses  to  support  renegotiation  of  current  and  future  increment  plans  and 
contents 

(d)  Validate  feasibility  and  cost-effectiveness  of  renegotiated  increment  plans  and  contents 

(e)  Monitor  effectiveness  of  configuration  and  version  management 


CSF  4.4  Identification  and  mitigation  planning  for  feasibility  evidence  shortfalls  and  other 

risks.  Ability  to  recommend  corrective  actions  for  feasibility  evidence  shortfalls  and 
other  risks 

(a)  Identify  and  evaluate  alternative  courses  of  action  to  address  feasibility  evidence  shortfalls, 
technical  risks,  and  root  causes 

(b)  Recommend  appropriate  corrective  actions  to  obtain  best-possible  system  outcomes 
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CSF  4.5  Use  of  milestone  reviews  to  ensure  stakeholder  commitment  to  proceed.  Ability  to: 

(a)  Prepare  plans,  schedules,  budgets,  scenarios,  and  facilities  for  evaluating  developer  feasibility 
evidence 

(b)  Identify  shortfalls  in  feasibility  evidence  as  program  risks 

(c)  Assess  developer  risk  management  plans  for  coverage  of  risks,  identify  shortfalls,  and 
recommend  corrective  actions 


Goal  5.  Professional  and  Interpersonal  Skills 

CSF  5.1  Leadership.  Ability  to  plan,  staff,  organize,  teambuild,  control,  and  direct  systems 
engineering  teams 

(a)  Prepare  top-level  plans,  schedules,  budgets,  and  deliverables  for  a  system  engineering  team 

(b)  Evaluate  and  recruit  appropriate  staff  members  for  executing  plans 

(c)  Involve  staff  members  in  collaborative  development  of  team  shared  vision,  detailed  plans,  and 
organizational  roles;  adjust  top-level  plans  as  appropriate 

(d)  Monitor  progress  with  respect  to  plans,  identify  shortfalls,  provide  mentoring  and  constructive 
corrective  actions  to  address  shortfalls 


CSF  5.2  Collaboration.  Ability  to  work  with  others  to  negotiate,  plan,  execute,  and 
coordinate  complementary  tasks  for  achieving  program  objectives 

(a)  Develop  understanding  of  other  participants’  value  propositions,  and  use  knowledge  to  negotiate 
mutually  satisfactory  roles,  responsibilities,  and  modes  of  collaboration 

(b)  Establish  modes  of  pro-active  coordination  of  emerging  issues  with  other  team  members  and 
teams 

(c)  Provide  help  to  others  in  need  of  your  capabilities 


CSF  5.3  Communication.  Ability  to  perform  timely,  coherent,  and  concise  verbal  and 
written  communication 

(a)  Develop  understanding  of  other  participants’  knowledge  boundaries  and  terminology,  and  adjust 
your  terminology  to  facilitate  their  understanding  of  your  communications 


57 


(b)  Provide  timely,  coherent,  and  concise  verbal  and  written  communication  within  your  team  and 
among  external  stakeholders 

(c)  Explore  new  low-tech  (wallboards)  and  high-tech  (wikis,  blogs,  videos)  modes  of  effective 
communications 


CSF  5.4  Accountability.  Ability  to  deliver  on  promises  and  behave  ethically 

(a)  Commit  to  and  follow  through  on  promised  commitments;  provide  advance  warning  of  potential 
delays  and  shortfalls 

(b)  Respect  the  truth,  intellectual  property,  and  the  rights  and  concerns  of  others 


CSF  5.5  Adaptability  and  Leaning.  Ability  to  cope  with  uncertainty  and  unexpected 
developments,  and  to  seek  help  and  fill  relevant  knowledge  gaps 

(a)  Be  prepared  to  cope  with  inevitable  uncertainty  and  unexpected  developments 

(b)  Identify  key  knowledge  and  skills  needed  for  your  project  and  career,  and  engage  in  learning 
activities  to  master  them 
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APPENDIX  B:  SEPAT  EXAMPLE  -  LOGISTICS  SUPPORT  SYSTEM 

PROJECT 
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Impact 


Evidence/Risk 


Question 

# 


E 


.c  ^ 


bo  <u 
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NOTE:  Impact  and  evidence/risk  ratings  should  be  done  independently.  The  impact 

rating  should  estimate  the  effect  a  failure  to  address  the  specified  item  might  have 

on  the  program.  The  evidence  rating  should  specify  the  qualtity  of  evidence  that  has 

been  provided,  which  demonstrates  that  the  specified  risk  item  has  been  Risk 

satisfactorily  addressed.  Exposure 


Rationale/ 
Source  of  evidence 


Goal  1: 


Concurrent  definition  of  system  requirements  and  solutions 


Critical  Success  Factor  1.1 


Understanding  of  stakeholder  needs:  capabilities,  operational  concept,  key 
performance  parameters,  enterprise  fit  (legacy) 

At  Milestone  A,  have  the  KPPs  been  identified  in  clear,  comprehensive,  concise  terms 
that  are  understandable  to  the  users  of  the  system? 

Has  a  CONOPS  been  developed  showing  that  the  system  can  be  operated  to  handle 
both  nominal  and  off-nominal  workloads,  and  to  meet  response  time  requirements? 

Has  the  ability  of  the  system  to  meet  mission  effectiveness  goals  been  verified  through 
the  use  of  modeling  and  simulation? 

Have  the  success-critical  stakeholders  been  identified  and  their  roles  and 
responsibilities  negotiated? 

Have  questions  about  the  fit  of  the  system  into  the  stakeholders'  context  --  acquirers, 
end  users,  administrators,  interoperators,  maintainers,  etc.  --  been  adequately 
explored? 


No  formal  Milestone  A 

IT  system  sized  using  vendor  benchmarks  and  expected  number  of  users 

IT  system  designed  to  replace  legacy  system  and  manual  processes.  Mission 
effectiveness  of  system  not  a  major  concern  during  development. 

Development  of  system  had  been  attempted  by  other  companies  and  failed. 
Stakeholders  had  been  previous  identified  and  were  involved  early  on. 

Explored  across  all  areas  early  on.  However,  new  sponsor  IT  PM  (assigned  to 
project  after  system  acceptance  but  before  deployment)  changed  system 
requirements  related  to  database  system  and  there  was  no  funding  left  to 
migrate  to  a  different  DBMS. 


Critical  Success  Factor  1.2 


Concurrent  exploration  of  solution  opportunities;  analysis  of  alternatives 
(AoAs)  for  cost-effectiveness  and  risk  (measures  of  effectiveness) 

Have  at  least  two  alternative  approaches  been  explored  and  evaluated? 

At  Milestone  B,  has  the  government  structured  the  program  plan  to  ensure  that  the 
contractor  addresses  the  allocation  of  capabilities  to  hardware,  software,  and  human 
elements  sufficiently  early  in  the  development  program? 

Has  the  claimed  degree  of  reuse  been  validated? 

Have  the  claimed  quality  of  service  guarantees  been  validated? 

Have  proposed  COTS  and  third-party  solutions  been  validated  for  maturity, 
compatibility,  supportability,  suitability,  and  effectiveness,  throughout  the  expected 
system  lifetime? 


Was  recommended,  but  rejected  by  sponsor  PM  due  to  "color  of  money' 
(legacy  replacement  needed  to  remain  on  upgraded  legacy  platform) 


No  planned  reuse  (unless  you  consider  use  of  GUI  builder  "reuse".  In  this 
case,  none  initially  planned,  but  change  in  plans  resulted  in  the  use  of  GUI 
Vendor  platform  benchmarks  used 

GUI  prototype  developed  and  demo'd  prior  to  finalizing  the  decision  to  use 
the  GUI  builder. 


Critical  Success  Factor  1.3 


System  scoping  &  requirements  definition  (external  interfaces;  memoranda 
of  agreement) 

Have  external  interface  complexities  been  identified  and  minimized  via  MoAs  or  their 
equivalent?  Is  there  a  plan  to  mitigate  their  risks? 

At  Milestone  B,  are  the  major  system-level  requirements  (including  all  KPPs)  defined 
sufficiently  to  provide  a  stable  basis  for  the  development  through  IOC? 

By  Milestone  A,  is  there  a  plan  to  have  information  exchange  protocols  established  for 
the  whole  system  and  its  segments  by  Milestone  B? 

Have  the  key  stakeholders  agreed  on  the  system  boundary  and  assumptions  about  its 
environment? 


Not  clear  why  this  was  yellow  based  on  response  to  first  subitem  (High 
Impact,  Good  Evidence)— Also  stayed  yellow  after  response  to  2nd  subitem 

External  interfaces  well  defined  and  used  by  legacy  system  being  replaced. 


N/A-no  formal  Milestone  A  and  system  used  well-defined,  existing  protocols 
for  external  interfaces. 

Based  on  current  business  processes  and  documented  in  system 
requirements 
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Critical  Success  Factor  1.4 


Prioritization  of  requirements  &  allocation  to  increments 

Can  an  initial  capability  be  achieved  within  the  time  that  the  key  program  leaders  are 
expected  to  remain  engaged  in  their  current  jobs  (normally  less  than  5  years  or  so 
after  Milestone  B)?  If  this  is  not  possible  for  a  complex  major  development  program, 
can  critical  subsystems,  or  at  least  a  key  subset  of  them,  be  demonstrated  within  that 
time  frame? 

At  Milestone  B,  do  the  requirements  take  into  account  likely  future  mission  growth 
over  the  program  life  cycle? 

Have  appropriate  early  evaluation  phases,  such  as  competitive  prototyping,  been 
considered  or  executed  for  high-risk/low-maturity  components  of  the  system? 

Have  stakeholders  agreed  on  prioritization  of  system  features  and  their  allocation  to 
development  increments? 


This  was  not  initially  perceived  as  a  high  impact,  but  turned  out  to  be  a  high 
impact.  System  was  developed,  accepted,  and  entered  OT&E  under  a  single 
program  leader,  but  leader  was  replaced  prior  to  deployment  and 
subsequent  leader  decided  that  she  did  not  want  to  deploy  the  system  since 
it  did  not  use  her  "preferred"  DBMS. 

Some  initial  problems  during  OT&E  due  to  the  fact  that  developers  had  not 
anticipated  how  often  users  would  resend  transactions  and  users  had 
request  that  ALL  transactions  be  saved  online.  Quick  fix  initiated  during 
OT&E  when  storage  capacity  of  system  exceeded. 


Single  increment  on  fixed  price  contract. 


Goal  2: 


System  life-cycle  organization,  planning,  and  staffing 


Critical  Success  Factor  2.1 


Establishment  of  stakeholder  life-cycle  responsibilities,  authorities,  and 
accountabilities  (RAAs)  (for  system  definition  &  system  development) 

Are  the  stakeholders  who  have  been  identified  as  critical  to  the  success  of  the  project 
represented  by  highly  qualified  personnel  —  those  who  are  collaborative, 
representative,  authorized,  committed,  and  knowledgeable? 

At  Milestone  A,  are  there  validated  plans,  budgets,  and  schedules  defining  how  the 
pre-Milestone  B  activity  will  be  done,  and  by  whom? 

Has  the  government  attempted  to  align  the  duration  of  the  program  manager's 
assignment  with  key  deliverables  and  milestones  in  the  program? 

Have  the  key  stakeholders  agreed  to  the  proposed  assignments  of  system  roles, 
responsibilities,  and  authorities? 


System  developed  for  small  organization  and  all  key  users  and  associated 
managers  identified  as  stakeholders  and  actively  participated  in  the 
development  process. 

No  formal  Milestone  A 

PM  lasted  through  acceptance  testing  and  into  OT&E 

Initial  assessed  impact  low,  but  turned  out  to  be  high  when  users  realized 
that  their  job  was  at  risk  since  the  new  system  automated  much  of  their  job, 
resulting  in  them  sabotaging  the  deployment  of  the  system  (along  with  the 
new  PM). 


Critical  Success  Factor  2.2 


Establishment  of  integrated  product  team  (IPT)  RAAs,  cross-IPT  coordination 
needs 

Does  the  project  make  effective  use  of  Integrated  Project  Teams  (IPTs)  throughout  the 
supplier  hierarchy? 

Are  the  IPTs  staffed  by  highly  qualified  personnel,  as  in  2.a(a)? 

For  IPTs  addressing  strongly  coupled  objectives,  are  there  "super-IPTs"  for  resolving 
conflicts  among  the  objectives? 


Critical  Success  Factor  2.3 


Establishment  of  necessary  plans  and  resources  for  meeting  objectives 


Have  decisions  about  the  use  of  one-shot,  incremental,  or  evolutionary  development 
been  validated  for  appropriateness  and  feasibility,  and  accepted  by  the  key 
stakeholders? 


Have  system  definition,  development,  test,  and  evolution  plans,  budgets,  and 
schedules  been  validated  for  appropriateness  and  feasibility,  and  accepted  by  the  key 
stakeholders? 

Is  there  a  valid  business  case  for  the  system,  relating  the  life  cycle  system  benefits  to 
the  system  total  cost  of  ownership? 


Critical  Success  Factor  2.4 


Establishment  of  appropriate  source  selection,  contracting,  and  incentive 
structures 

Has  the  competitive  prototyping  option  been  addressed,  and  the  decision  accepted  by 
key  stakeholders? 

If  doing  competitive  prototyping,  have  adequate  plans  and  preparations  been  made 
for  exercising  and  evaluating  the  prototypes,  and  for  sustaining  core  competitive 
teams  during  evaluation  and  down  selection? 


Is  the  status  of  the  contractor's  business  and  team  "healthy,"  both  in  terms  of  business 
indicators,  and  within  the  industrial  base  for  the  program  area?  Is  the  program  aligned 
with  the  core  business  of  the  unit,  and  staffed  adequately  and  appropriately? 


Has  the  acquiring  organization  successfully  completed  projects  similar  to  this  one  in 
the  past? 

Has  the  candidate  performing  organization  successfully  completed  projects  similar  to 
this  one  in  the  past? 

Is  the  program  governance  process,  and  in  particular  the  system  engineering  plan,  well 
articulated  and  compatible  with  the  goals  of  the  program? 


Critical  Success  Factor  2.5 


Establishment  of  necessary  personnel  competencies 

Does  the  government  have  access  over  the  life  of  the  program  to  the  talent  required 
to  manage  the  program?  Does  it  have  a  strategy  over  the  life  of  the  program  for  using 
the  best  people  available  in  the  government,  the  FFRDCs,  and  the  professional  service 
industry? 
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N/A— small  agile-like  team 

N/A 

N/A 


Project  designed  to  be  single  increment  FFP  contract.  When  development 
team  brought  on  and  began  to  validate  initial  plans,  realized  that  the  project 
was  not  do-able  and  investigated  alternatives,  resulting  in  an  Ada  waiver, 
the  use  of  a  GUI  builder,  and  a  small  "agile"  development  team  (that  also 
produced  2167A  deliverables). 

Not  clear  what  better  evidence  would  be.  Initial  plans  adjusted  and 
presented  to  key  stakeholders  at  requirements  review,  PDR,  and  CDR,  with 
prototype  demo  presented  by  PDR.  COCOMO  cost  model  not  useful  due  to 
nature  of  development  using  GUI  builder  (pre-COCOMO  II) 
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N/A--system  developed  using  known  technologies  and  products 


Contractor's  established  team  at  company  headquarters  started  the 
development,  then  transitioned  development  to  the  customer's  location 
after  hiring  new  staff  for  the  local  office.  New  hires  were  well-vetted  and 
intially  worked  with  headquarter's  team  through  a  transition  period.  Not 
clear  what  other  evidence  one  would  look  for. 

This  project  did  not  need  to  employ  "rocket  science".  It  was  a  fairly  typically 
IT  system  with  some  key,  but  well-understood,  external  interfaces. 

This  project  did  not  need  to  employ  "rocket  science".  It  was  a  fairly  typically 
IT  system  with  some  key,  but  well-understood,  external  interfaces. 

No  SEP  was  required  for  this  progam  (well  below  the  MDAP  threshold) 


Yellow  risk  button  not  enabled  here.  In  response  to  questions,  sponsor  PM 
was  a  computer  scientist  with  IT  experience. 


At  Milestone  B,  have  sufficiently  talented  and  experienced  program  and  systems 
engineering  managers  been  identified?  Have  they  been  empowered  to  tailor 
processes  and  to  enforce  development  stability  from  Milestone  B  through  IOC? 


Has  the  government  attempted  to  align  the  duration  of  the  program  manager's 
assignment  with  key  deliverables  and  milestones  in  the  program? 

Is  the  quantity  of  systems  engineering  personnel  assigned,  their  skill  and  seniority  mix, 
and  the  time  phasing  of  their  application  throughout  the  program  life  cycle 
appropriate? 


Potentially  high  impact  due  to  the  FFP  nature  of  the  contract.  Not  sure  what 
"good"  or  "externally  validated"  evidence  is  here.  People  selected  in  the 
headquarter's  office  were  "known  quantities"  within  the  development 
organization  and  staffed  the  project  with  a  new  team  at  the  customer's 
location.  New  staff  were  interviewed  and  references  checked.  However,  I 
would  not  call  this  "externally  validated"  since  it  is  known  that  references 
are  hesitant  to  say  anything  negative  due  to  the  fear  of  lawsuits. 

Not  a  major  issue  since  the  program  was  only  scheduled  for  about  18 
months. 

Not  really  a  major  issue  for  an  IT  application  using  COTS  products. 


Goal  3: 


Technology  maturing,  architecting 


Critical  Success  Factor  3.1 


COTS/NDI/Services  evaluation,  selection,  validation  for  maturity  & 
compatibility 

Have  COTS/NDI/Services  opportunities  been  evaluated  prior  to  baselining 
requirements? 


Have  COTS/NDI/Services  scalability,  compatibility,  quality  of  service,  and  life  cycle 
support  risks  been  thoroughly  addressed? 


Has  a  COTS/NDI/Services  life  cycle  refresh  strategy  been  developed  and  validated? 


Proposed  COTS  were  established  products 

At  the  time  the  products  were  selected,  they  were  probably  adequately 
evaluated.  However,  a  few  years  later,  ORACLE  drove  several  other  DBMS 
products  out  of  the  market  place.  Not  clear  that  could  have  been 
anticipated  at  the  time  since  ORACLE  was  going  through  some  growing  pains 
at  the  time  this  project  was  initiated.  In  addition,  options  were  limited  due 
to  the  "color  of  money"  and  ORACLE  was  not  thought  to  be  a  candidate 
since  the  legacy  system  was  not  built  upon  ORACLE. 

Seemed  reasonable  at  the  time  since  an  established  vendor  was  used. 
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1 


2 
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Critical  Success  Factor  3.2 


Life-cycle  architecture  definition  &  validation 

Has  the  system  been  partitioned  to  define  segments  that  can  be  independently 
developed  and  tested  to  the  greatest  degree  possible? 

By  Milestone  A,  is  there  a  plan  to  have  internal  and  external  information  exchange 
protocols  established  for  the  whole  system  and  its  segments  by  Milestone  B? 

Does  the  project  have  adequate  processes  in  place  to  define  the  verification,  test  & 
validation,  and  acceptance  of  systems  and  system  elements  at  all  phases  of  definition 
and  development? 

Is  there  a  clear,  consistent,  and  traceable  relationship  between  system  requirements 
and  architectural  elements?  Have  potential  off-nominal  architecture-breakers  been 
addressed? 

Does  the  architecture  adequately  reconcile  functional  hardware  "part-of"  hierarchies 
with  layered  software  "served-by"  hierarchies? 

Has  a  Work  Breakdown  Structure  (WBS)  been  developed  with  the  active  participation 
of  all  relevant  stakeholders,  which  accurately  reflects  both  the  hardware  and  the 
software  product  structure? 


Basis  for  work  assignments  across  the  programming  team. 

Used  existing  protocols  implemented  in  legacy  system 

The  "evidence"  seemed  reasonable  since  subject  matter  experts  from  a 
subcontractor  organization  were  used  and  the  prime  supported  the 
development  of  the  test  plan  and  procedure  documentation. 

Not  really  an  issue  with  this  IT  system 

Not  really  an  issue  with  this  IT  system 

Not  really  an  issue  with  this  IT  system-all  hardward  was  standard  COTS 


Use  of  prototypes,  exercises,  models,  and  simulations  to  determine 
technological  solution  maturity 


Will  risky  new  technology  mature  before  Milestone  B?  Is  there  a  risk  mitigation  plan? 


Have  the  key  non-technical  risk  drivers  been  identified  and  covered  by  risk  mitigation 
plans? 

Is  there  a  sufficient  collection  of  models,  and  appropriate  simulation  and  exercise 
environments,  to  validate  the  selected  concept  and  the  CONOPS  against  the  KPPs? 

Has  the  claimed  degree  of  reuse  been  validated? 


Contract  was  awarded  at  Milestone  B  with  plan  to  use  no  new  technologies. 
When  it  was  decided  to  use  a  new-on-the-market  GUI  builder  after  contract 
award,  the  tool  was  tested  during  the  development  of  the  user  l/F  prototype 
before  a  final  decision  was  made. 


Not  really  an  issue  with  this  IT  system.  The  user  l/F  was  prototyped  and 
evaluated  by  the  key  users  prior  to  completion  of  PDR. 

No  reuse  was  initially  planned  (is  this  a  duplicate  question?) 


Critical  Success  Factor  3.4 


Validated  system  engineering,  development,  manufacturing,  operations  & 
maintenance  budgets  and  schedules 

Are  the  major  known  cost  and  schedule  drivers  and  risks  explicitly  identified,  and  is 
there  a  plan  to  track  and  reduce  uncertainty? 

Have  the  cost  confidence  levels  been  developed  and  accepted  by  the  key  system 
stakeholders? 

Is  there  a  top-to-bottom  plan  for  how  the  total  system  will  be  integrated  and  tested? 
Does  it  adequately  consider  integration  facilities  development  and  earlier  integration 
testing? 

If  time-boxing  or  time-determined  development  is  used  to  stabilize  schedules,  have 
features  been  prioritized  and  the  system  architected  for  ease  of  adding  or  dropping 
borderline  features? 

Are  there  strategies  and  plans  for  evolving  the  architecture  while  stabilizing 
development  and  providing  continuity  of  services? 


High  impact  due  to  FFP  nature  of  contract,  but  were  identified  and  managed 
well  early-on 

Total  cost  was  negotiated  prior  to  contract  award. 

Development  lab  provided  at  contractor's  facility  that  supported  both 
development  and  test.  This  was  established  prior  to  development  contract 
award. 

N/A  due  to  single  increment 

Not  really  an  issue  with  an  IT/DBMS-based  application.  What  might  have 
been  more  of  an  issue  was  the  underlying  data  model,  but  most  data 
elements,  tables,  user  forms  were  well  defined  early-on. 


Goal  4: 


Evidence-based  progress  monitoring  and  commitment  reviews 


Critical  Success  Factor  4.1 


Monitoring  of  system  definition  &  development  progress  vs.  plans  2 


Are  the  levels  and  formality  of  plans,  metrics,  evaluation  criteria,  and  associated 
mechanisms  (IMP,  IMS,  WBS,  EVMS)  commensurate  with  the  level  of  project 
requirements  emergence  and  stability?  (Too  little  is  risky  for  pre-specifiable  and  stable 
requirements;  too  much  is  risky  for  emergent  and  unstable  requirements.) 

Are  the  project's  staffing  plans  and  buildup  for  progress  monitoring  adequate  with 
respect  to  required  levels  of  expertise? 

Have  most  of  the  planned  project  personnel  billets  been  filled  with  staff  possessing  at 
least  the  required  qualification  level? 

Is  the  project  adequately  identifying  and  managing  its  risks? 

Have  the  processes  for  conducting  reviews  been  evaluated  for  feasibility, 
reasonableness,  completeness,  and  assurance  of  independence? 

Has  compliance  with  legal,  policy,  regulatory,  standards,  and  security  requirements 
been  clearly  demonstrated? 


Level  of  formality  may  have  been  excessive  for  project.  However,  due  to  the 
FFP  nature  of  the  contract,  the  development  organization  used  corporate 
standards  for  risky  programs  to  monitor  this  program. 


2 

2 

1 

2 

2 

1 

2 

2 

2 

2 

2 

2 


Critical  Success  Factor  4.2 


Monitoring  of  feasibility  evidence  development  progress  vs.  plans  2 

Has  the  project  identified  the  highest  risk  areas  on  which  to  focus  feasibility  analysis?  GUI  environment 

Has  the  project  analyzed  alternative  methods  of  evaluating  feasibility  (models, 

simulations,  benchmarks,  prototypes,  reference  checking,  past  performance,  etc.)  and  See  above 

prepared  the  infrastructure  for  using  the  most  cost-effective  choices? 

Has  the  project  identified  a  full  set  of  representative  operational  scenarios  across 
which  to  evaluate  feasibility? 

Has  the  project  prepared  milestone  plans  and  earned  value  targets  for  measuring 
progress  in  developing  feasibility  evidence? 

Is  the  project  successfully  monitoring  progress  and  applying  corrective  action  where 
necessary? 


Critical  Success  Factor  4.3 


Monitoring,  assessment,  and  replanning  for  changes  in  needs,  opportunities, 
and  resources 


Does  the  project  have  an  effective  strategy  for  performing  triage  (accept,  defer,  reject) 
on  proposed  changes,  which  does  not  destabilize  ongoing  development? 


Strategy  defined,  but  seldom  used  on  project 


Does  the  project  have  an  adequate  capability  for  performing  change  impact  analysis 
and  involving  appropriate  stakeholders  in  addressing  and  prioritizing  changes? 

Is  the  project  adequately  verifying  and  validating  proposed  changes  for  feasibility  and 
cost-effectiveness? 


Process  detined,  but  seldom  used  on  project 


Only  received  a  couple  of  change  requests  during  project 


Critical  Success  Factor  4.4 


Use  of  milestone  reviews  to  ensure  stakeholder  commitment  to  proceed  2 

Are  milestone  review  dates  based  on  the  availability  of  feasibility  evidence,  instead  of 
the  availability  of  artifacts  or  the  occurrence  of  planned  review  dates? 

Are  artifacts  and  evidence  of  feasibility  evaluated,  and  risky  shortfalls  identified,  by 
key  stakeholders  and  independent  experts,  prior  to  review  events? 

Are  developer  responses  to  identified  risks  prepared  prior  to  review  events? 

Do  reviews  achieve  risk-based  concurrence  of  key  stakeholders  on  whether  to  proceed 

into  the  next  phase?  (Proceed;  skip  a  phase;  revisit  the  current  phase;  terminate  or  Not  an  option  after  FFP  contract  issued 

rescope  the  project.) 


APPENDIX  C:  SECAT  EXAMPLE  -  LOGISTICS  SUPPORT  SYSTEM 

PROJECT 


66 


01 

3 

O 
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2 
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Question 

# 

Goal  1: 

Critical 

l-l(a) 

1.1(b) 

1.1(c) 

1.1(d) 

1.1(e) 

Critical 

1.2(a) 

1.2(b) 

1.2(c) 

1.2(d) 

Critical 

1.3(a) 

1.3(b) 

1.3(c) 


Impact 


Competency/Risk 


NOTE:  Impact  and  evidence/risk  ratings  should  be  done  independently.  The  impact 

rating  should  estimate  the  effect  a  failure  to  competently  address  the  specified  item 

might  have  on  the  program.  The  competency  rating  should  specify  the  observed, 

historical  experience  and  competency  of  the  systems  engineering  staff  on  past  Risk 

programs  with  respect  to  the  specified  risk  item.  Exposure 


Concurrent  definition  of  system  requirements  and  solutions 


Success  Factor  1.1 


Understanding  of  stakeholder  needs:  capabilities,  operational  concept,  key 
performance  parameters,  enterprise  fit  (legacy).  Ability  to  analyze  strengths  2 
and  shortfalls  in  current-system  operations  via: 

Participatory  workshops,  surveys,  focus  groups 

Operations  research  techniques:  operations  data  collection  and  analysis 
Mission  effectiveness  modeling  and  simulation 
Prototypes,  scenarios,  stories,  personas 

Ethnographic  techniques:  Interviews,  sampled  observations,  cognitive  task  analysis 


Success  Factor  1.2 


Concurrent  exploration  of  solution  opportunities;  analysis  of  alternatives 
(AoAs)  for  cost-effectiveness  and  risk  (measures  of  effectiveness).  Ability  to 
identify  and  assess  alternative  solution  opportunities  via  experimentation 
and  analysis  of: 

Alternative  work  procedures,  non-materiel  solutions 

Purchased  or  furnished  products  and  services  Limited  by  color  of  money 

Emerging  technology  Limited  by  color  of  money 

Competitive  prototyping  Not  prior  to  contract  award. 


Success  Factor  1.3 


System  scoping  &  requirements  definition  (external  interfaces;  memoranda 
of  agreement).  Ability  to  establish  system  scope  and  requirements  via: 

Cost-schedule-effectiveness  assessment  of  needs  vs.  opportunities 

Organizational  responsibilities,  authorities,  and  accountabilities  (RAAs)  assessment 

Appropriate  degrees  of  requirements  completeness,  consistency,  testability,  and 
variability  due  to  emergence  considerations 


Rationale/ 
Source  of  evidence 


Project  too  small 


Critical  Success  Factor  1.4 


Prioritization  of  requirements  &  allocation  to  increments.  Ability  to  prioritize 
requirements  and  schedule  them  into  increments  based  on  considerations  of: 

Stakeholder  priorities  and  returns  on  investment 

Capability  interdependencies  and  requirements  emergence  considerations 
Technology  maturity  and  implementation  feasibility  risks 


Goal  2: 


System  life-cycle  organization,  planning,  and  staffing 


Incremental  delivery  not  an  option.  Project  too  small. 


Critical  Success  Factor  2.1 


Establishment  of  stakeholder  life-cycle  responsibilities,  authorities,  and 
accountabilities  (RAAs)  (for  system  definition  &  system  development).  Ability 
to  support  establishment  of  stakeholder  RAAs  via  conduct  of: 

Organizational  capability  analyses 

Stakeholder  negotiations 

Operational  exercise  analyses 


Well  established  in  sponsor  IT  group 


Critical  Success  Factor  2.2 


Establishment  of  integrated  product  team  (IPT)  RAAs,  cross-IPT  coordination 
needs.  Ability  to  establish  IPT  RAAs  and  cross-IPT  coordination  mechanisms 
via: 

Risk  identification,  analysis,  and  prioritization 
Organizational  RAAs  and  skills  availability  assessment 
Risk  interdependency  analysis 
Risk  resolution  cost-benefit  analysis 


Not  really  done  due  to  size  of  project,  small  team  co-located,  with 
considerable  inputs  from  stakeholders. 


Critical  Success  Factor  2.3 


Establishment  of  necessary  resources  for  meeting  objectives.  Ability  to 
support  program  negotiation  of  objectives  vs.  resources  via: 


Cost-schedule-capability  tradeoff  analyses 

Use  of  requirements  priorities  and  interdependencies  to  support  negotiation  of 
increment  contents 

Development  of  strategies  to  adjust  increment  content  to  meet  delivery  schedules 

Analysis  of  project  change  traffic  and  rebaselining  of  future-increment  plans  and 
specifications 


Not  much  negotiation  after  award  of  FFP  contract.  However,  some 
2  negotions  conducted  in  order  to  incorporate  new  higher  priority 

requirement  and  drop  lower  priority  requirements  for  a  no-cost  change. 


No  future  increments  planned 


Critical  Success  Factor  2.4 


Establishment  of  appropriate  source  selection,  contracting,  and  incentive 
structures.  Ability  to  support  program  management  in  preparing  source 
selection  materials,  matching  contracting  and  incentive  structures  to  program 
objectives,  and  technical  monitoring  of  performance  vs.  program  objectives: 

Preparation  of  proposal  solicitation  materials  and  evaluation  capabilities  and 
procedures 

Evaluation  of  proposal  submissions  with  respect  to  criteria 

Technical  support  of  contract  negotiations 

Technical  support  of  contract  performance  monitoring 


Critical  Success  Factor  2.5 


Assurance  of  necessary  personnel  competencies.  Ability  to  support  program 
management  in  evaluating  proposed  staffing  plans  and  monitoring  staffing 
capabilities  vs.  plans  in  the  areas  of: 

Concurrent  definition  of  system  requirements  &  solutions 

System  life  -cycle  organization,  planning,  and  staffing 
Technology  maturing  and  architecting 
Evidence-based  progress  monitoring  &  commitment  reviews 
Professional  and  interpersonal  skills 


N/A  due  to  size  of  project.  Project  done  under  an  "umbrella"  services 
contract. 


Good  staff  on  project,  but  system  requirements/solution  approach  already 
established  by  contract  award.  Probably  done  in  large  part  by  the 
government. 


Goal  3: 


Technology  maturing,  architecting 


Critical  Success  Factor  3.1 


COTS/NDI  evaluation,  selection,  validation  for  maturity  &  compatibility. 
Ability  to  evaluate  alternative  combinations  of  COTS,  NDI,  and  purchased 
services  for: 

Functional  capabilities  vs.  system  needs 

Levels  of  service:  performance,  resilience,  scalability,  usability,  tailorability,  etc. 
Mutual  compatibility  and  external  interoperability 
Supplier  maturity,  stability,  support,  and  responsiveness 
Acquisition  and  operational  costs 


Key  was  identification  of  GUI  builder  to  eliminate  Ada  requirment  for 
interactive  database  application. 


Done  using  vendor  benchmarks.  Selection  of  vendor  limited  due  to  color  of 
money  for  replacement  system. 

COTS/NDI  not  an  issue  as  long  as  network  connectivity  allowed 


Not  clear  how  to  rate  this  given  above  constraints 


Critical  Success  Factor  3.2 


Life-cycle  architecture  definition  &  validation.  Ability  to  define  and  evolve 
configurations  of  hardware  and  software  components  and  connectors  along 
with  human  operational  architectures,  and  to  validate  that  they  cost- 
effectively  support  program  objectives: 

Define  candidate  hardware/software/human-operational  architectures 

Evaluate  their  functional  capabilities,  levels  of  service,  interoperability,  and 
sustainability  vs.  system  needs 

Perform  tradeoff  analyses  among  functional  capabilities  and  levels  of  service 


Only  one  hardware  configuration  based  on  COTS 


Critical  Success  Factor  3.3 


Done  with  respect  to  GUI  builder 

Assess  relative  costs,  schedules,  and  capabilities  of  various  combinations  of  evaluation 
methods 

Prepare  plans  for  enabling  and  performing  evaluations 


Use  of  prototypes,  exercises,  models,  and  simulations  to  determine 
technological  solution  maturity.  Ability  to  assess  the  relative  costs  and 
benefits  of  alternative  evaluation  methods,  and  apply  appropriate 
combinations  of  methods: 


Prepare  representative  nominal  and  off-nominal  scenarios,  workload  generators, 
virtual  component  surrogates,  and  testbeds  to  support  evaluations 

Perform  evaluations,  analyze  results,  investigate  anomalies,  and  adjust  plans  as 
appropriate 


Not  reasonable  for  small  project 


Not  done  with  any  rigor  due  to  size  of  project/schedule 


Critical  Success  Factor  3.4 


Validated  system  engineering,  development,  manufacturing,  operations  & 
maintenance  budgets  and  schedules.  Ability  to: 

Assess  alternative  budget  and  schedule  estimation  methods  vs.  nature  of  system, 
degree  of  system  knowledge,  complementarity  of  estimates,  and  cost  vs.  accuracy  of 
performing  estimates 

Prepare  plans  for  gathering  inputs  and  performing  estimates 

Perform  selected  combinations  of  estimates  and  reconcile  their  differences 

Perform  tradeoff  analyses  among  functional  capabilities,  levels  of  service,  costs,  and 
schedules 


Manufacturing  n/a.  Operations  not  within  scope  of  contract-government  to 
assume  operations  and  maintenance  at  end  of  contract  using  existing  staff 
maintaining/operating  system  being  replace. 


Not  an  option 


Goal  4: 


Evidence-based  progress  monitoring  and  commitment  reviews 


Critical  Success  Factor  4.1 


Monitoring  of  system  definition,  development,  &  test  progress  vs.  plans. 
Ability  to  plan,  monitor,  and  evaluate  technical  progress  vs.  plans: 

Prepare  test  &  evaluation  facilities  &  plans  and  define  data  to  be  provided  for 
assessing  technical  progress  vs.  project  plans 

Monitor  performers'  technical  progress  in  developing,  verifying  and  validating  their 
technical  solutions 

Identify  shortfalls  in  technical  progress  vs.  plans,  and  determine  their  root  causes 


Critical  Success  Factor  4.2 


4.2(a) 

4.2(b) 


Monitoring  of  feasibility  evidence  development  and  test  progress  vs.  plans. 
Ability  to: 

Evaluate  developers'  feasibility  evidence  assessment  and  test  plans  for  coverage,  cost- 
effectiveness,  and  realism  of  assumptions 

Monitor  developers'  progress  with  respect  to  plans,  identify  shortfalls  and  root  causes 
Evaluate  feasibility  evidence  produced,  identify  shortfalls  and  root  causes 


4.2(c) 


Critical  Success  Factor  4.3 


Monitoring,  assessment,  and  replanning  for  changes  in  needs,  opportunities, 
and  resources.  Ability  to: 

Assess  proposed  changes  in  program  objectives,  constraints,  plans,  and  resources 

Perform  triage  to  determine  which  should  be  handled  immediately,  deferred  to  future 
increments,  or  rejected 

Perform  tradeoff  analyses  to  support  renegotiation  of  current  and  future  increment 
plans  and  contents 

Validate  feasibility  and  cost-effectiveness  of  renegotiated  increment  plans  and 
contents 

Monitor  effectiveness  of  configuration  and  version  management 


Identification  and  mitigation  planning  for  feasibility  evidence  shortfalls  and 
other  technical  risks.  Ability  to  recommend  corrective  actions  for  feasibility 
evidence  shortfalls  and  technical  risks: 

Identify  and  evaluate  alternative  courses  of  action  to  address  feasibility  evidence 
shortfalls,  technical  risks,  and  root  causes 

Recommend  appropriate  corrective  actions  to  obtain  best-possible  system  outcomes 


Critical  Success  Factor  4.5 


Use  of  milestone  reviews  to  ensure  stakeholder  commitment  to  proceed. 
Ability  to: 

Prepare  plans,  schedules,  budgets,  scenarios,  tools,  and  facilities  for  evaluating 
developer  feasibility  evidence 

Identify  shortfalls  in  feasibility  evidence  as  program  risks 

Assess  developer  risk  management  plans  for  coverage  of  risks,  identify  shortfalls,  and 
recommend  corrective  actions 


Goal  5: 


Professional  and  interpersonal  skills 


Critical  Success  Factor  5.1 


Leadership.  Ability  to  plan,  staff,  organize,  teambuild,  control,  and  direct 
systems  engineering  teams. 

Prepare  top-level  plans,  schedules,  budgets,  and  deliverables  for  a  system  engineering 
team 

Evaluate  and  recruit  appropriate  staff  members  for  executing  plans 

Involve  staff  members  in  collaborative  development  of  team  shared  vision,  detailed 
plans,  and  organizational  roles;  adjust  top-level  plans  as  appropriate 
Monitor  progress  with  respect  to  plans,  identify  shortfalls,  provide  mentoring  and 
constructive  corrective  actions  to  address  shortfalls 


Critical  Success  Factor  5.2 


Collaboration.  Ability  to  work  with  others  to  negotiate,  plan,  execute,  and 
coordinate  complementary  tasks  for  achieving  program  objectives 

Develop  understanding  of  other  participants'  value  propositions,  and  use  knowledge 
to  negotiate  mutually  satisfactory  roles,  responsibilities,  and  modes  of  collaboration 

Establish  modes  of  pro-active  coordination  of  emerging  issues  with  other  team 
members  and  teams 

Provide  help  to  others  in  need  of  your  capabilities 


Critical  Success  Factor  5.3 


Communication.  Ability  to  perform  timely,  coherent,  and  concise  verbal  and 
written  communication 

Develop  understanding  of  other  participants'  knowledge  boundaries  and  terminology, 
and  adjust  your  terminology  to  facilitate  their  understanding  of  your  communications 

Provide  timely,  coherent,  and  concise  verbal  and  written  communication  within  your 
team  and  among  external  stakeholders 

Explore  new  low-tech  (wallboards)  and  high-tech  (wikis,  blogs,  videos)  modes  of 
effective  communications 


Critical  Success  Factor  5.4 


Accountability.  Ability  to  deliver  on  promises  and  behave  ethically 


Commit  to  and  follow  through  on  promised  commitments;  provide  advance  warning 
of  potential  delays  and  shortfalls 

Respect  the  truth,  intellectual  property,  and  the  rights  and  concerns  of  others 


Adaptability  and  Learning.  Ability  to  cope  with  uncertainty  and  unexpected 
developments,  and  to  seek  help  and  fill  relevant  knowledge  gaps 

Be  prepared  to  cope  with  inevitable  uncertainty  and  unexpected  developments 

Identify  key  knowledge  and  skills  needed  for  your  project  and  career,  and  engage  in 
learning  activities  to  master  them 


General  comment  on  tool  overall:  Too  detailed  and  too  subject  to 
interpretation.  It  depends  on  who  is  going  to  do  the  evaluation  and  what 
criteria  they  are  using.  It  also  depends  upon  the  size  and  scope  of  the 
project.  For  example,  it  will  be  difficult  for  someone  who  does  not  already 
know  the  staff  to  evaluate  accountability...  and  an  organization  is  not 
typically  going  to  divulge  the  fact  that  they  don't  think  all  of  their  staff  are 
accountable/ethical. 
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x  =  covered  by  EM 

SERC  EM  Task  Coverage  Matrix  VI. 0 

(x)  =  partially  covered  (unless  stated  otherwise) 

NRC 

Probability  of 

Success 

SE  Leading 
Indicators 

LIPSF 

(Stevens) 

Anchoring 

SW  Process 

(USC) 

PSSES 

(U.  of  Alabama) 

SSEE 

(CMU/SEI) 

Macro  Risk 
Model/Tool 

Concept  Dev 

1 

Atleast  2  alternatives  have  been  evaluated 

X 

X 

X 

X 

(w.r.t  NPR) 

(x) 

2 

Can  an  initial  capability  be  achieved  within  the 
time  that  the  key  program  leaders  are  expected 
to  remain  engaged  in  their  current  jobs 
(normally  less  than  5  years  or  so  after  Milestone 
B)?  If  this  is  not  possible  for  a  complex  major 
development  program,  can  critical  subsystems, 
or  at  least  a  key  subset  of  them,  be 
demonstrated  within  that  time  frame? 

X 

(x) 

X 

X 

(5  years  is  not 
explicitly 
stated) 

(x) 

(seems  to  be 
inferrable  from 
the  conclusions) 

(x) 

(implies  this) 

3 

Will  risky  new  technology  mature  before  B?  Is 
there  a  risk  mitigation  plan? 

X 

X 

X 

(x) 

X 

X 

4 

Have  external  interface  complexities  been 
identified  and  minimized?  Is  there  a  plan  to 
mitigate  their  risks? 

X 

X 

X 

X 

X 

X 

KPP  and  CONOPS 

5 

At  Milestone  A,  have  the  KPPs  been  identified  in 
clear,  comprehensive,  concise  terms  that  are 
understandable  to  the  users  of  the  system? 

X 

(x) 

X 

(X) 

x 

(strongly 

implied) 

(X) 

(implied) 

X 

X 

6 

At  Milestone  B,  are  the  major  system-level 
requirements  (including  all  KPPs)  defined 
sufficiently  to  provide  a  stable  basis  for  the 
development  through  IOC? 

X 

X 

(X) 

X 

X 

(x) 

(X) 

(There  is  no 
direct  reference 

to  this  but  is 
inferrable) 

X 

7 

Has  a  CONOPS  been  developed  showing  that  the 
system  can  be  operated  to  handle  the  expected 
throughput  and  meet  response  time 
requirements? 

X 

X 

(X) 

(X) 

X 

(x) 

(there  is  a  mention 
of  a  physical 
solution.  That's  the 

closest  in  this 

regard) 

X 

X 

NRC 

Probability  of 

Success 

SE  Leading 

Indicators 

LIPSF 

(Stevens) 

Anchoring 

SW  Process 

(USC) 

PSSES 

(U.  of  Alabama) 

SSEE 

(CMU/SEI) 

Macro  Risk 
Model/Tool 

Cost  and  Schedule  Scoping 

8 

Are  the  major  known  cost  and  schedule  drivers 
and  risks  explicitly  identified,  and  is  there  a  plan 
to  track  and  reduce  uncertainty? 

X 

X 

(x) 

X 

(x) 

(seems  to  imply) 

(x) 

(They  aren't 
identified  per 
se.  It's  only 
"questioned"  as 
such.) 

X 

9 

Has  the  cost  confidence  level  been  accepted  by 
the  stakeholders  for  the  program? 

X 

X 

X 

X 

(x) 

X 

(not  directly 
stated) 

(x) 

10 

Is  there  a  sufficient  collection  of  models  and  an 
appropriate  simulation  environment  to  validate 
the  selected  concept  and  the  CONOPS  against 
the  KPPs? 

X 

X 

X 

X 

(x) 

(x) 

(seems  to  be) 

11 

At  Milestone  B,  do  the  requirements  take  into 
account  likely  future  mission  growth  over  the 
program  life  cycle? 

X 

X 

(X) 

(X) 

(x) 

X 

Architecture  dev 

12 

Has  the  system  been  partitioned  to  define 
segments  that  can  be  independently  developed 
and  tested  to  the  greatest  degree  possible? 

X 

(x) 

(not  directly 
stated  as 
such) 

X 

X 

13 

By  Milestone  A,  is  there  a  plan  to  have 
information  exchange  protocols  established  for 
the  whole  system  and  its  segments  by  Milestone 
B? 

X 

(x) 

Seems  far  fetched 
though 

(X) 

X 

14 

At  Milestone  B,  has  the  government  structured 
the  program  plan  to  ensure  that  the  contractor 
addresses  the  decomposition  of  requirements  to 
hardware  and  software  elements  sufficiently 
early  in  the  development  program? 

X 

(x) 

(nothing  specific  to 
the  contractor 
though) 

X 

Risk  Assessment 

15 

Have  the  key  risk  drivers  (not  only  the 
technology  drivers)  been  identified? 

X 

X 

X 

(X) 

Indirect 

inkling 

X 

(x) 

(majorly  technical) 

X 

X 

NRC 

Probability  of 

Success 

SE  Leading 

Indicators 

LIPSF 

(Stevens) 

Anchoring 

SW  Process 

(USC) 

PSSES 

(U.  of  Alabama) 

SSEE 

(CMU/SEI) 

Macro  Risk 
Model/Tool 

Program  Implementation  Strategy 

16 

Does  the  government  have  access  over  the  life 
of  the  program  to  the  talent  required  to  manage 
the  program?  Does  it  have  a  strategy  over  the 
life  of  the  program  for  using  the  best  people 
available  in  the  government,  the  FFRDCs,  and 
the  professional  service  industry? 

X 

(x) 

(x) 

X 

17 

At  Milestone  A,  is  there  a  plan  defining  how  the 
pre-Milestone  B  activity  will  be  done,  and  by 
whom? 

X 

(x) 

X 

X 

(x) 

18 

Is  there  a  top-level  plan  for  how  the  total  system 
will  be  integrated  and  tested? 

X 

X 

(x) 

X 

X 

X 

X 

19 

At  Milestone  B,  have  sufficiently  talented  and 

experienced  program  and  systems  engineering 
managers  been  identified?  Flavethey  been 
empowered  to  tailor  processes  and  to  enforce 
requirements  stability  from  Milestone  B  through 
IOC? 

X 

(X) 

X 

X 

(x) 

(loosely 

connected) 

(x) 

X 

20 

Flas  the  government  attempted  to  align  the 
duration  of  the  program  manager's  assignment 
with  key  deliverables  and  milestones  in  the 
program? 

X 

Miscellaneous 

21 

Status  of  Contractor's  business  and  his  team? 
Corporate  (about  the  organization)  and  or 
program  indicators  ( team  set  up  and  issues 
faced  etc). 

X 

X 

22 

Program  fit  in  capability  vision 

X 

(x) 

(X) 

23 

Staffing  and  manning  status/planning 

X 

X 

X 

X 

24 

Process  Compliance  (or  rationale  of  choice) 

(X) 

(allows  for 

process 

tailoring) 

X 

X 

(x) 

(X) 

(X) 

25 

Review  of  review  process 

(x) 

(Sharing  best 
practices  with 
other  agencies) 

(x) 

Pretty  close 

X 

(x) 

X 

(X) 

26 

Reuse  claim  validation 

(x) 

X 

X 

27 

Level  of  Service  Validation 

X 

(X) 

NRC 


Probability  of 
Success 


SE  Leading 
Indicators 


28 

COTS  and  third  party  solutions 

X 

29 

Technical  success/progress  measurement 

(x) 

X 

30 

Verification/Validation  and  Configuration 
management 

X 

(V&V  seems 
implicit) 

31 

What  phases  of  the  SDLC  would  be  included  in 
developing  the  system 

(x) 

(implied  in 

process 

tailoring) 

32 

Using  Integrated  Project  Teams  (IPT) 

33 

Correlation  with  success  on  previous  projects 

34 

Frequency  of  change  in  requirements 

(x) 

(Indirect 
reference  in 

CM) 

35 

About  the  contract  (contract  change  orders 
received,  %age  subcontracted  to  suppliers, 
current/initial  contract  value  of  project) 

X 

36 

Stakeholder  Identification 

37 

Questions  about  the  product  -  Who  is  acquiring 
this  product,  end  users,  Ul,  environment  of 
use/deployment  etc. 

38 

Compliance  with  policy/standards/security  etc 

39 

Requirements/Architecture  Trace 

(x) 

40 

Funding  Stability 

41 

Key  reviews  slip  more  than  30  days 

42 

Program  governance  process,  System  Eng.  Plan 
well  articulated 

(x) 

X 

43 

Process  Improvement 

X 

(X) 

(mentioned  as  a 
part  of  process 
tailoring  effort) 

44 

Work  Breakdown  structure 

(x) 

(mentions  about 
work  products) 

LIPSF 

(Stevens) 

Anchoring 

SW  Process 

(USC) 

X 

PSSES 

(U.  of  Alabama) 

SSEE 

(CMU/SEI) 

X 

Macro  Risk 
Model/Tool 

X 

X 

X 

X 

X 

(X) 

X 

X 

X 

(X) 

X 

X 

X 

X 

X 

(X) 

X 

(X) 

X 

X 

X 

X 

X 

X 

(X) 

X 

(X) 

X 

X 

X 

X 

X 

(X) 

(implied  in 

process 

strategies) 

X 

X 

(X) 

(mention  about 
incremental 
development  of 
work  products) 
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Systems  Engineering  Effectiveness  Measures  Framework 

DAPS  Methodology  v2.0 

1  =  Mission  Capabilites;  2  =  Resources;  3  =  Management; 

4  =  Technical  Process;  5  =  Performance;  6  =  Special  Interest  Areas 

General  Comment:  To  be  truly  complementary  to  the  DAPS,  the  questions  need  to  be  put  in  perspective  of  the  program  life  cycle.  Are  these 
questions  focused  on  the  contractor/supplier  or  the  government  acquisition  team?  It  is  often  difficult  to  map  because  if  it  unclear  what  is 
appropriate  when.  Also  need  to  look  at  terminology.  Often,  the  terminology  is  too  vague  to  understand  what  is  really  intended  by  the  question. 

SE  EM  Framework  Area 

Interpretation 

DAPS 

Section 

DAPS  Topic  Covered 

Com  men  ts/ Observations 

References* 

*Not  every  reference  is  recorded  here. 

4  1  Concurrent  definition  of  system  requirements  and 

solutions 

5 

6 

1.1 

Understanding  of  stakeholder  needs:  capabilities, 
operational  concept,  key  performance  parameters, 
enterprise  fit  (legacy) 

1.1 

CONOPS 

7 

1. 

1. 

1 

At  Milestone  A,  have  the  KPPs  been  identified  in  clear, 
comprehensive,  concise  terms  that  are  understandable 
to  the  users  of  the  system? 

Understandable, 

comprehensive 

requirements 

1.3.2 

KPPs  and  KSAs 

KPPs  are  required  to  be  "established  and 
documented";  no  guidance  on 
understandability/quality  of  KPP;  however 

"1.3. 2. Q7:  After  reviewing  the  table  with  the  program's  KPPs,  including  Net-Ready  and  Force 
Protection  KPPs,  can  the  PMO  personnel  explain  the  rationale  for  the  thresholds  and  objectives?" 

8 

1. 

1. 

2 

Has  a  CONOPS  been  developed  showing  that  the  system 
can  be  operated  to  handle  both  nominal  and  off- 
nominal  workloads,  and  to  meet  response  time 
requirements? 

feasible  workload 
demonstrated  at  CONOPS; 
are  scenarios  quantified 

1.1 

1.2 

1.3.1 

CONOPS 

Analysis  of  Alternatives 

Reasonableness,  Stability,  and  Testability 

DAPS  specifies  "realistic"  scenarios,  not 
necessarily  "nominal  and  off-nominal 
workloads."  These  terms  and  "response 
time"  are  more  network-oriented. 
Quantification  is  given  by  "measures  of 
effectiveness." 

"The  material  from  a  CONOPS  will  feed  into  many  elements  of  information  required  by  the 
Department  of  Defense  (DoD),  such  as  the  JCIDS  process,  the  Test  and  Evaluation  (T&E)  process, 
and  the  Analysis  of  Alternatives  (AoA). 

1.2.1. Q1:  How  were  mission  tasks  (MTs),  measures  of  effectiveness  (MOEs),  and  measures  of 
performance  (MOPs)  derived  from  relevant  guidance  on  requirements  or  capabilities  (e.g.. 

Mission  Needs  Statement  (MNS),  Operational  Requirements  Document  (ORD)  (if  pertinent),  or  the 
problem  statement  found  in  the  ICD?  [1.2.1.C1] 

•Are  they  quantifiable?  [1.2.1.C1] 

1.2.1. Q5:  Are  the  threats  and  scenarios  realistic  and  current?  [1.2.1. Cl] 

1.3.1. Q5:  How  does  the  ICD  describe  the  threats  and  the  operational  environment  in  which  the 
capabilities  are  to  be  exercised? 

Were  the  threats  and  scenarios  validated  by  the  Defense  Intelligence  Agency  (DIA)?  [1.3.1.C1] 

9 

1. 

1. 

3 

Has  the  ability  of  the  system  to  meet  mission 
effectiveness  goals  been  verified  through  the  use  of 
modeling  and  simulation? 

verification  of  mission 
goals 

1.2.1 

1.3.1 

4.4.2 

Validity  and  Currency  (1.2  Analysis  of 
Alternatives) 

Reasonableness,  Stability,  and  Testability 
Modeling  and  Simulation  Tools 

1.2.1. Q2:  Are  the  MOEs  stated  in  terms  of  military  utility  and  based  on  value  provided  to  the 
warfighter? 

•Are  these  MOEs  used  to  identify  models,  simulations,  and  other  analysis  tools  required  to 
execute  the  study?  [1.2.1.C1] 

1.3.1. C12:  Verification  of  all  KPPs,  MOEs,  measures  of  suitability  (MOSs),  and  Critical  Technical 
Parameters  (CTPs)  are  demonstrated  by  prototypes  or  engineering  development  models  operating 
in  the  system's  intended  environment.  Results  are  documented  in  test  and  evaluation  reports 
described  and  documented  in  accordance  with  the  Test  and  Evaluation  Master  Plan  (TEMP). 
Deficiencies  have  been  documented  and  analyzed,  and  the  associated  risks  for  successful  testing 
are  manageable. 

4.4.2. C1:  The  program  has  a  documented  modeling  and  simulation  (M&S)  approach  for  design  and 
analysis,  which  covers  its  purpose  and  use.  All  assumptions  and  weaknesses  inherent  in  the 
program's  M&S  activities  are  made  apparent  to  decision  makers.  This  approach  is  cross- 

10 

1. 

1. 

4 

Have  the  success-critical  stakeholders  been  identified 
and  their  roles  and  responsibilities  negotiated? 

stakeholders  identifed  and 

roles  defined 

3.3.3 

Management  Structure  and  Communications 

DAPS  does  not  address  the  "negotiation"  of 
roles  and  responsibilities.  This  term  implies 
that  the  person  performing  the  task  has  the 
option  to  choose  not  to  perform  some  part  of 
the  task.  In  DAPS,  the  PMO  is  in  charge  and 
identifies  what  needs  to  be  done  and  who 

shall  address  it. 

3.3.3. C3:  The  PMO  is  organized  to  execute  the  SDD  phase.  Program  IPTs  or  equivalent  are  formed 
and  will  include  all  appropriate  program  stakeholders  to  support  SDD  (ideally  these  IPTs  are  jointly 
formed  with  the  contractor  IPTs).  The  organization  includes  support  from  the  acquisition 
organization  infrastructure,  agencies  like  DCMA,  OSD,  and  from  contracted  support  personnel,  as 
required.  The  roles  and  responsibilities  are  clearly  defined  and  consistent  with  achieving  program 
objectives. 

3.3.3. C5:  The  contractor  development  team  is  organized  to  execute  the  SDD  phase.  Program  IPTs 
or  equivalent  are  formed  and  include  representatives  from  all  appropriate  stakeholders,  including 
the  PMO.  The  team  includes  support  from  the  company  infrastructure,  subcontractors  and 
contracted  support  personnel,  as  required.  Roles,  responsibilities,  and  lines  of  authority  are  clearly 
defined  and  consistent  with  achieving  program  objectives. 

11 

1. 

1. 

5. 

Have  questions  about  the  fit  of  the  system  into  the 
stakeholders'  context  -  acquirers,  end  users, 
administrators,  interoperators,  maintainers,  etc.  --  been 
adequately  explored? 

system  fit  in  context; 
scenarios  cover  spectrum 
of  occurrences  once 

fielded 

1.3 

4.1.6 

Capabilities:  Perspective 

Sustainment  as  a  Design  Consideration 

"For  a  materiel  solution  to  a  capability  requirement,  fielding  an  operational  capability  starts  with 
sound  strategies  for  requirements,  acquisition,  test  and  evaluation  (T&E),  and  support  and 
sustainment.  To  be  viable,  these  strategies  will  be  developed  in  concert  and  require  early  and 
ongoing  collaboration  among  operators,  developers,  acquirers,  testers,  sustainers,  and  operations 
analysts.  No  one  strategy  can  stand  alone  and  still  be  viable  because  all  are  interdependent  and 
require  the  integration  of  the  others  to  be  effective." 

"4.1.6. Q4:  How  is  the  program  planning  to  include  inputs  from  warfighters,  users,  developers, 
acquirers,  technologists,  testers,  budgeters,  and  sustainers  during  capability  needs  development?" 

12 

13 

1.2 

Concurrent  exploration  of  solution  opportunities; 
analysis  of  alternatives  (AoAs)  for  cost-effectiveness 

1.2 

Analysis  of  Alternatives 

(c)  2009  Systems  Engineering  Research  Center 


Page  1 


Working  Copy 


SE  EM  Framework  Area 

Interpretation 

DAPS 

DAPS  Topic  Covered 

Comments/Observations 

References* 

Section 
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14 

1. 

Have  at  least  two  alternative  approaches  been  explored 

alternative  approaches 

1.2 

Analysis  of  Alternatives 

Inferred,  by  definition  of  AoA 

2. 

1 

and  evaluated? 

15 

1. 

At  Milestone  B,  has  the  government  structured  the 

allocation  of  capabilities  to 

4.5.1 

Software  Development  Plan 

Pre-milestone  B:  4.5.1.C8:  Externally  visible  properties  of  the  system,  manifested  in  software  and 

2. 

program  plan  to  ensure  that  the  contractor  addresses 

physical  architecture 

hardware,  have  resulted  in  requirements  and  architecture  artifacts  that  have  been  carried  forward 

2 

the  allocation  of  capabilities  to  hardware,  software,  and 

from  Milestone  A  and  resulted  in  plans  and  technical  data  that  are  driving  requirements 

human  elements  sufficiently  early  in  the  development 

refinement,  design,  and  test  development. 

program? 

4.5.1.Q9:  Walkthrough  the  architecture  and  design  of  the  system  as  known  now,  and 
demonstrate  alignment  between  program  office  and  contractor  views.  Focus  particularly  on 
requirements  development  and  traceability,  identifying  artifacts  and  processes  that  demonstrate 
ongoing  alignment  among  the  program  office  and  contractor,  as  requirements  evolve  from 
externally  visible  (architecture)  properties  to  internally  visible  (design)  properties.  [4.5.1.C8] 

16 

1. 

Has  the  claimed  degree  of  reuse  been  validated? 

Actual  reuse,  reuse 

3.3.4 

Management  Methods,  metrics,  and  Techniques 

3.3.4.3.C8:  The  Government  Program  Office  should  initially  approve  the  program  metrics  and  then 

2. 

validated 

Software  Development  Plan 

periodically,  e.g.,  monthly,  the  metrics  should  be  reported  and  reviewed.  These  metrics  should 

3 

4.5.1 

include  many,  if  not  all  of  the  following:  Development  status  S  curves;  Processor  throughput 
utilization;  Processor  memory  utilization;  Input/output  utilization;  Software  Engineering  Staffing; 
Software  Work  Packages  Summary;  Schedule  Performance  Index;  Cost  performance  Index; 
Problem/Deficiencies  /Discrepancies  Status;  Requirements  Stability;  Software  Size;  Software 

Reuse  Status  (planned  versus  'actuals');  Reliability  Growth  Curve;  Logistics  Footprint  Reduction; 
Planned  Operational  Effectiveness;  Product  Availability  Predictions;  O&S  Cost  Projections; 
Development  Test  entrance  criteria  and  status;  DAES  Reporting  (For  MDAPS);  Milestone  B  and  C 
entrance  criteria. 

4.5. 1Q8:  Does  the  Software  Development  Plan  provide  for  early  demonstrations  (prior  to  the 
Preliminary  Design  Review  (PDR))  of  software  reuse  candidates  on  system  simulations?  [4.5.1.C7 
Reuse  of  software,  from  existing  systems  or  prior  development  efforts,  has  been  analyzed  for  com| 

17 

1. 

Have  the  claimed  quality  of  service  guarantees  been 

validated  system 

4.4.2 

Modeling  &  Simulation  Tools 

DAPS  does  not  use  the  term  "quality  of 

4.4.2.C3:  The  program  uses  M&S  during  the  Concept  Refinement  and  Technology  Development 

2. 

validated? 

performance 

5.1 

Performance  Effectiveness 

service";  assume  it  is  the  same  as  system 

phases  to: 

4 

performance  parameters 

...Identify  and  assess  the  system's  performance  in  its  intended  operating  environment  -  both 
physical  (mechanical  and  electromagnetic)  and  operational  (information  exchange,  threat,  etc.) 
environments 

5.1.1.C6:  Sufficient  CTPs  are  identified  in  the  TEMP  and  measure  critical  system  characteristics 
that,  when  achieved,  allow  the  attainment  of  desired  operational  performance  capabilities.  With 
each  technical  parameter,  thresholds  are  identified  for  each  stage  of  development. 

18 

1. 

Have  proposed  COTS  and  third-party  solutions  been 

COTS  maturity,  suitability 

1.2.1 

Validity  and  Currency 

Does  not  address  COTS  "validation" 

1.2.1.Q9:  Does  the  prioritized  list  resulting  from  the  AMA  address  technological  maturity. 

2. 

validated  for  maturity,  compatibility,  supportability. 

5.4.1 

Assessed  Manufacturing 

specifically,  but  does  require  a  determination 

technological  risk,  supportability,  and  the  affordability  of  each  approach  using  the  best  available 

5 

suitability,  and  effectiveness,  throughout  the  expected 

of  suitabilty  through  an  acceptance  process. 

data  in  the  pre-ICD  process?  [1.2.1.C2] 

system  lifetime? 

5.4.1. C12:  Planned  non-developmental  items  (NDI)  or  commercial-off-the-shelf  (COTS)  items  have 
been  determined  to  meet  program  system  performance  and  sustainment  requirements  through  a 
defined  acceptance  process. 

5.4.1. Q12:  What  are  the  NDI  or  COTS  items  being  used  in  the  TD? 

•  What  are  the  sources  of  these  items? 

•  How  have  these  items  been  determined  to  meet  intended  program  performance  requirements? 
[5.4.1.C12] 

5.4.1. C24:  Planned  NDI  and  COTS  items  have  been  determined  to  meet  program  system 
performance  and  sustainment  requirements  through  a  defined  acceptance  process. 

5.4.1. C43:  Planned  NDI  or  COTS  items  have  been  determined  to  meet  program  system 

19 

20 

1.3 

System  scoping  &  requirements  definition  (external 

3.3.6 

Management  of  Dependencies  and  External 

3.3.6.C11:  The  boundary  and  scope  of  the  SoS  is  understood  by  the  PM  and  system  engineers  and 

interfaces;  memoranda  of  agreement) 

Interfaces  (FoS  /  SoS) 

the  SoS  is  adaptable  to  boundary  and  scope  changes  over  time.  All  systems  included  in  the  SoS 
should  be  identified.  Interfaces  from  the  SoS  to  external  systems  should  be  defined  and  scoped. 
Specific  stakeholders  of  the  SoS  and  its  systems  should  be  identified,  including  their  organization. 
Identification  of  the  users  for  each  system  is  key. 

(c)  2009  Systems  Engineering  Research  Center 
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21 

1. 

3. 

1 

Have  external  interface  complexities  been  identified 
and  minimized  via  MoAs  or  their  equivalent?  Is  there  a 
plan  to  mitigate  their  risks? 

External  interface 

agreements 

3.3.1 

3.3.6 

Program  Plan/Schedule 

Management  of  Dependencies  and  External 
Interfaces  (FoS  /  SoS) 

3.3.1.Q32:  How  does  the  program  ensure  that  all  key  strategies  and  top-level  plans  remain 
consistent  and  aligned  (i.e.,  coordinated)  with  the  IMP/IMS? 

•Are  the  type  and  number  of  technical  reviews  correct  in  each  appropriate  plan? 

•Does  the  IMS  capture  both  the  government  SEP  and  the  prime  contractor's  SEMP/SEP  activities, 
events,  and  milestones? 

•  Are  the  scheduled  interfaces  w  FoS/SoS  correctly  captured  in  the  IMS,  SEP,  TEMP,  and  other 
related  plans? 

•  Did  the  plans  adequately  address  or  reference  all  key  processes  (e.g..  Requirements,  Risk 
Management,  V&V,  Monitoring  &  Control,  Continuous  process  improvement,  etc.)?  [3.3.1.C5] 

3.3.6. Q5:  Does  the  SEP  and  TES  address  the  interface  interdependency  plans  for  development  and 
test.  [3.3.6.C6] 

3.3.6. Q14:  How  will  FoS/SoS  interfaces  be  managed?  And  what  is  the  plan  to  resolve  issues  that 
cross  PM,  PEO,  and  Service  lines? 

•  Have  Interface  Control  Documents  been  identified/developed  and  Interface  Control  Working 
Groups  been  assigned? 

•  Provide  a  summary  of  the  Memorandums  of  Agreement  (MOAs) 

•  Do  the  MOAs  include  any  "triggers"  that  require  a  FoS/SoS  member  to  inform  the  others  if  there 

3.3.6. C11:  The  boundary  and  scope  of  the  SoS  is  understood  by  the  PM  and  system  engineers  and  1 

22 

1. 

3. 

2 

At  Milestone  B,  are  the  major  system-level  requirements 
(including  all  KPPs)  defined  sufficiently  to  provide  a 
stable  basis  for  the  development  through  IOC? 

requirement  definition 
level  sufficient  for 
development 

4.2.1 

Analysis  and  Decomposition  (Pre  Milestone  B 
criteria) 

4.2.1. C7:  System  requirements  specifications  and  performance  test/verification  requirements  are 
linked  and  verification  methods  are  defined.  Note:  Allocation  of  system  functions  defines  the 
functional  baseline  of  the  system  design. 

•  Traceability  to  current  requirements  documentation  is  configuration  managed  for  approved 
capability  upgrades  commensurate  with  maturity  of  the  technology  required  for  the  upgrade. 
Maturity  is  verified  through  readiness  assessments  and  well-defined  metrics. 

•The  system  architecture  is  well  defined  and  documented,  and  is  in  accordance  with  all  applicable 
standards,  protocols  and  data  interchange  definitions  as  defined  by  key  interface  descriptions. 
•Test  verification  descriptions,  critical  to  the  process,  are  defined  for  each  performance 
requirement. 

•Specifications  are  allocated  and  defined  to  the  appropriate  level  consistent  with  the  System 
Development  and  Demonstration  (SDD)  phase  objectives. 

4.2.1. C12:  ‘Software  requirements  are  evaluated  to  ensure  that  they  are  complete,  unambiguous, 
correct,  consistent,  verifiable,  modifiable,  traceable,  ranked  for  importance,  and  ranked  for  stabilit 
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1. 

3. 

3 

By  Milestone  A,  is  there  a  plan  to  have  information 
exchange  protocols  established  for  the  whole  system 
and  its  segments  by  Milestone  B? 

Communication  protocols 

3.3.3 

Management  Structure  and  Communications 
(pre-Milestone  A  criteria) 

Management  Structure  and  Communications 
(pre-Milestone  B  criteria) 

3.3.3. C2:  The  PMO  organization  is  structured  to  interface  closely  and  openly  with  the  contractor 
as  well  as  other  stakeholder  organizations.  The  PMO  leverages  other  government  organizations  to 
benefit  the  TD  effort. 

3.3.3. C6:  The  contractor  program  office  communicates  programmatic  information  internally  and 
externally  in  a  timely  and  accurate  manner  across  the  contract  team  including  subcontractors.  For 
large,  geographically  distributed  system  development,  electronic  database  tools  are  used  to 
support  this  communication.  The  participating  groups  and  functions,  including  production  and 
support  functions,  are  tied  into  the  communication  channels  and  process. 
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1. 

3. 

4 

Have  the  key  stakeholders  agreed  on  the  system 
boundary  and  assumptions  about  its  environment? 

Stakeholder  agreement  on 
requirements 

4.2.2 

3.2.2 

3.3.6 

4.2.4' 

Management  of  Requirements 

Entrance  and  Exit/Success  Criteria 

Management  of  Dependencies  and  External 
Interfaces 

Trade  Studies  and  Approaches 

DAPS  seems  to  give  the  mechanisms  to  do  so, 
but  does  not  always  ask  if  agreement  has 
been  achieved;  it  seems  to  be  assumed  by 
the  collaborative  efforts. 

4.2.2. Q8:  Do  the  stakeholders  understand  and  accept  all  the  requirements?  [4.2.2.C3] 

3.2.2. C5:  Exit/Success  Criteria  from  CR  phase  -  all  reviews,  technical  and  programmatic  (i.e.,  ITR 
and  ASR),  in  support  of  specific  decision  points  have  been  successfully  conducted  with  valid 
documentation,  data  and  analyses. 

3.2.2. Q21:  As  a  result  of  the  ASR,  does  the  resulting  set  of  requirements  agree  with  the  customer 
needs  and  expectations,  and  can  the  system  under  review  proceed  into  the  TD  phase?  [3.2.2. C5] 
3.3.6.C12:  In  a  SoS  program,  the  technical  planning  process  must  be  initiated  top-down  but 
iterated  within  individual  systems  until  a  consensus  approach  is  agreed  upon  and  resourced. 
Systems  engineers  from  across  the  SoS  must  share  data  and  plans  and  engage  as  part  of  a 
collaborative  team  for  the  SoS.  It  is  important  to  recognize  the  value  of  a  collaborative  SE  team 
and  value  of  integration  facilities,  which  promote  open  and  active  exchange  and  experimentation 
among  members  of  the  SoS  SE  team. 

4.2.4.C2:  The  trade  space  (i.e.,  the  set  of  program  and  system  parameters,  attributes,  and 

characteristics  required  to  satisfy  performance  standards) 

has  been  identified  in  general  terms  and  agreed  to  by  the  stakeholders  - 

the  program  manager  (PM)  and  the  capability  needs  approval  authority. 
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DAPS  often  mentions  an  incremental  4.2.2.C5:  The  program's  systems  engineering  (SE)  process  during  the  Technology  Development 

approach  but  really  does  not  mention  (TD)  phase  is  disciplined  in  documenting  and  tracking  specifications  at  all  levels,  and  structured  to 

anything  on  how  to  divide  the  requirements  manage  changes.  Integral  to  this  process  is  configuration  management  (CM).  The  CM  plan  lays  out 
into  increments  or  to  prioritize  requirements,  the  process  and  plans  to  ensure  that  designs  are  traceable  to  requirements,  that  change  is 
It  seems  to  assume  that  all  requirements  will  controlled  and  documented,  that  interfaces  are  defined  and  understood,  and  that  there  is 
be  implemented  as  planned.  consistency  between  the  product  and  its  supporting  documentation. 

4.2.2.Q9:  How  does  the  requirements  management  plan  address  the  validation  of  requirements? 

•  How  are  the  prioritized  evaluation  criteria  consistent  with  requirements,  and  the  operations  and 
sustainment  concepts?  [4.2.2. C3] 


DAPS  does  not  address  the  timeframe  of  3.1.1.Q1:  How  is  the  Acquisition  Strategy  realistic? 

personnel  assignments,  just  feasibility  within  •  How  are  the  program  objectives  attainable? 

the  program's  schedule  •  What  is  the  strategic  approach  to  attaining  the  program  objectives? 

•  Can  this  strategic  approach  be  successfully  implemented  with  reasonable  certainty?  Note:  There 
is  no  simple  formula  for  ensuring  the  approach  is  realistic.  To  evaluate  it,  reviewers  must  perform 
a  detailed  study  of  the  threat,  assess  the  state-of-the-art  in  all  technology  areas,  review  past 
performance  on  similar  acquisitions  or  systems,  and  survey  industry  capability,  then  attain 
consensus  on  the  complete  analysis.  Studies  take  time  and  resources,  but  because  realism  is  such 
an  important  criterion  for  a  successful  strategy,  every  effort  should  be  made  to  support  this 
undertaking  in  critical  areas  [3.1.1.C1] 

Requires  "flexibility"  for  change  4.2.1.Q21:  What  are  the  features  of  the  design  architecture  that  will  ensure  it  remains  robust  and 

adaptable  throughout  the  system  life  cycle?  [4.2.1. C7] 

4.2.2. C4:  The  evolutionary  Acquisition  Strategy  (AS)  utilizes  a  management  system  that  continually 
defines  the  requirements  and  development  activities  to  support  the  evolving  needs;  adequately 
addresses  the  various  concerns  of  users,  developers,  and  managers;  and  mitigates  the  risks 
associated  with  these  issues  are  mitigated.  The  basic  system  architecture  is  designed  to 
accommodate  change.  Techniques  such  as  open  systems  design,  functional  partitioning  and 
modular  design  have  been  addressed  by  the  PM  to  achieve  a  flexible  system  that  can  be  easily 

_ and  affordably  modified. _ 

Emphasis  is  on  identifying  the  risks  and  does  Perspective:  Experienced  program  personnel  provide  data  regarding  critical  and  high-risk  efforts 
not  say  specificially  how  to  handle  the  risks,  and  identify  as  realistically  as  possible  the  expected  schedule,  which  the  program  management 
"Prototypes  are  used  as  part  of  the  SDD  office  then  compares  with  the  top-level  defense  program  schedule  template  to  determine  the 
process,  as  are  reviews,  methods,  and  tools."  actual  schedule  risk  and  to  identify  all  schedule  drivers.  With  this  approach,  the  probability  of 
overrunning  a  program  schedule  can  be  estimated  by  determining  how  much  risk  exists  and 
where  it  is  greatest.  This  approach  enables  program  managers  (PMs)  to  estimate  early  and 
continuously  in  the  program  the  possibility  of  a  significant  likelihood  of  overrunning  the  program 
schedule  by  determining  how  much  and  where  the  risk  to  successful  schedule  completion  is 
greatest. 

"Early  industry  involvement  is  essential  in  the  identification  of  the  critical  and  high-risk  efforts  in 
the  development  of  the  integrated  schedule.  Integrated  scheduling  describes  the  detailed  tasks 
that  support  the  significant  activities  identified  in  integrated  planning  and  timing  of  tasks." 
Identifies  highest  risk  path  and  ensures  PM  is  applying  resources  on  it. 

3.3.1. Q21:  How  are  programs  with  high  risk  shown  in  the  IMS  in  order  to  give  the  visibility 
to  manage  and  control  risk?  [3.3.1.C2a] 

4.1.1. C3:  Pending  the  next  version  of  DoDI  5000.2,  "3. 5.2.6.  A  list  of  known  or  probable 
Critical  Program  Information  (CPI)  and  potential  countermeasures  such  as  Anti-Tamper  (AT) 
in  the  preferred  system  concept  and  in  the  critical  technologies  and  competitive  prototypes 
to  inform  program  protection  (DoDD  5200.39,  Reference  (ai))  and  design  integration  during  in 
the  TD  phase." 

"The  use  of  competitive  prototyping  is  required  by  the  Office  of  the  Under  Secretary  of  Defense 
for  Acquisition,  Technology  and  Logistics  (OUSD(AT&L))  policy  through  the  Technology 
Development  phase  up  to  Milestone  B,  which  will  include  the  Preliminary  Design  Review." 

4.3.1. C13:  The  SRR  is  typically  held  well  in  advance  of  Milestone  B  to  allow  time  for  issue 
resolution  and  proper  executive  level  concurrence  on  process  and  results. 

4.3.3. C3:  The  functional  baseline  should  be  established  at  the  System  Functional  Review  (SFR) 
during  the  Technology  Development  phase.  Competitive  prototypes  of  system  or  subsystem 
components  should  be  developed  and  tested  to  ensure  program  requirements  are  achievable. 
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1. 

Have  stakeholders  agreed  on  prioritization  of  system 

Prioritization  of 

4.5.1 

Software  Development  Plan 

Ranking  of  requirements  is  only  mentioned  in 

4.5.1.Q4:  Walk  through  the  architecture  of  the  system  as  known  now,  and  demonstrate  alignment 

4. 

features  and  their  allocation  to  development 

capabilities/  requirements 

4.2.1 

(Requirements)  Analysis  and  Decomposition 

the  Software  Development  section. 

between  program  office  and  contractor  views.  Focus  on  requirements  traceability,  from  initial 

4 

increments? 

specification  of  capabilities  to  high-level  requirements  and  preliminary  architecture.  [4.5.1. C3] 

4.5.1. Q9:  Walkthrough  the  architecture  and  design  of  the  system  as  known  now,  and 
demonstrate  alignment  between  program  office  and  contractor  views.  Focus  particularly  on 
requirements  development  and  traceability,  identifying  artifacts  and  processes  that  demonstrate 
ongoing  alignment  among  the  program  office  and  contractor,  as  requirements  evolve  from 
externally  visible  (architecture)  properties  to  internally  visible  (design)  properties.  [4.5.1.C8] 

4.5.1. C11:  Software  process  integration  has  facilitated  timely  and  efficient  program  integration. 
Information  flow  has  not  been  impeded,  and  risks  traceable  to  information  flow  have  been 
perceived  and  mitigated  in  a  timely  fashion.  There  has  been  agreement  on  software  metrics  and 
plans  between  the  program  office  and  contractor. 

4.2.1. C12:  ‘Software  requirements  are  evaluated  to  ensure  that  they  are  complete, 
unambiguous,  correct,  consistent,  verifiable,  modifiable,  traceable,  ranked  for  importance,  and  rai 
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32 

2  System  life-cycle  organization,  planning,  and  staffing 

3.3.1 

Program  Plan/Schedule 

33 

34 

2.1 

Establishment  of  stakeholder  life-cycle  responsibilities, 

2.2.2 

Continuity  and  Stability 

2.2.2.Q3:  How  are  the  total  life  cycle  support  requirements  and  responsibilities  addressed  in  the 

authorities,  and  accountabilities  (RAAs)  (for  system 

3.2.2 

Knowledge-based  Decisions  and  Milestones 

Logistics  Support  Plan?  What  are  the  bases  for  the  estimates?  [2.2.2.C1] 

definition  &  system  development) 

(Entrance  and  Exit/Success  Criteria) 

3.2.2.Q30:  In  preparation  for  the  IBR,  were  the  following  documents  provided  by  the  contractor  to 

3.3.1 

Program  Plan/Schedule 

the  government  for  review? 

3.3.3 

Management  Structure  and  Communications 

•  Work  Authorization  Documents  (WADs) 

•  Responsibility  Assignment  Matrix  (RAM) 

3.3.1.Q33:  How  is  the  SEP  updated  and  used  by  the  Technical  Leads  and  PM  to  manage  the 
technical  aspects/efforts  of  the  program? 

•Was  the  SEP  prepared  in  time  to  support  RFPs? 

•Was  the  SEP  updated  after  contract  award  to  document  the  major  events,  revisions,  slips  in  the 

schedule,  technology  immaturity,  etc.  that  have  occurred?  Note:  The  SEP  is  the  PM's  overarching 
technical  management  tool  that  reflects  both  government  and  contractor  activities,  roles,  and 
responsibilities.  It  is  a  living  dynamic  plan,  updated  as  necessary  [3.3.1.C5] 

3.3.3. C5:  The  contractor  development  team  is  organized  to  execute  the  SDD  phase.  Program  IPTs 
or  equivalent  are  formed  and  include  representatives  from  all  appropriate  stakeholders,  including 
the  PMO.  The  team  includes  support  from  the  company 

infrastructure,  subcontractors  and  contracted  support  personnel,  as  required.  Roles, 
responsibilities,  and  lines  of  authority  are  clearly  defined  and  consistent  with  achieving 
program  objectives. 

3.3.3. Q9:  How  are  the  various  IPTs  organized  on  the  program,  and  do  they  have 
responsibility,  experienced  staff,  and  authority  to  make  decisions? 

35 

2. 

Are  the  stakeholders  who  have  been  identified  as  critical 

Qualified  staff 

2.3.1 

Sufficiency  of  Numbers  and  Qualifications 

DAPS  addresses  the  issue  of  quality  staff  in 

2.3.1.C1:  There  is  an  established  program/process  in  the  program  management  office  (PMO)  that 

1. 

to  the  success  of  the  project  represented  by  highly 

general,  but  does  not  differentiate  with 

provides  the  right  number  and  mix  of  qualified  personnel  to  successfully  execute  the  Technology 

1 

qualified  personnel  -  those  who  are  collaborative. 

critical  areas;  also  does  not  define  what 

Development  (TD)  phase.  There  is  sufficient  flexibility  in  the  program  to  address  program 

representative,  authorized,  committed,  and 

qualified  people  are. 

shortfalls  through  the  use  of  Systems  Engineering  Technical  Assistance  (SETA)  contractor 

knowledgeable? 

personnel. 

2.3.1.C2:  The  contractor  has  an  established  program  that  provides  the  right  number  and  mix  of 
qualified  personnel  to  successfully  execute  the  TD  phase.  Key  contractor  management  and 
technical  personnel,  including  the  program  manager,  chief  systems  engineer,  software  architect, 
and  functional  area  managers,  have  worked  successfully  on  projects  of  similar  complexity  and 
have  had  significant  work  experience  relevant  to  the  current  program  phase. 
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How  is  a  plan  "validated"?  3.3.1.C1:  The  Integrated  Master  Plan  (IMP)  is  an  event-driven  plan  that  documents  the  significant 

accomplishments  necessary  to  complete  the  work  and  ties  each  accomplishment  to  a  key 
program  event  that  forms  the  foundation  of  the  Integrated  Master  schedule  (IMS).  Note:  The  IMP 
Events  are  not  tied  to  calendar  dates;  each  event  is  completed  when  its  supporting 
Accomplishments  are  completed  and  as  evidenced  by  the  Criteria  completion  supporting  each  of 
those  Accomplishments. 

•Was  the  SEP  updated  after  contract  award  to  document  the  major  events,  revisions,  slips  in  the 
schedule,  technology  immaturity,  etc.  that  have  occurred?  Note:  The  SEP  is  the  PM's  overarching 
technical  management  tool  that  reflects  both  government  and  contractor  activities,  roles,  and 
responsibilities.  It  is  a  living  dynamic  plan,  updated  as  necessary  [3.3.1.C5] 

3.3.1. C6:  During  program  planning,  the  government  created  a  top  level  program  schedule  or 
Roadmap  which  provides  a  capstone  program  summary  allowing  insight  into  the  government's 
program  planning  and  approval  process.  This  initial  Roadmap,  along  with  other  program 
documentation,  provides  the  basis  for  an  initial  set  of  expectations  for  the  program  with  the  warfif 
•Is  prepared  by  the  government  program  office  early  in  the  program  planning  phase  in 
conjunction  with  any  other  supporting  or  associated  government  program  offices 

•  Is  focused  on  and  conveys  the  "big  picture"  of  the  program  objectives,  capabilities 
evolution,  summary  schedule,  and  any  major  program  constraints 
•Supports  initial  and  subsequent  budget  submissions  and  provides  the  basis  for 
developing  a  sound  position  on  funding  cuts  or  increases  throughout  the  program  life 
•Contains  key  events  and  shows  critical  schedule  interfaces  (e.g.,  IOC  and  FOC)  with 
all  supporting  programs  and  activities  (for  example,  other  Services,  DARPA,  and  other 
agencies)  and  their  supporting  contracts 

•Is  reviewed  regularly  by  the  primary  program  team  and  supporting  program  teams  to 
assess  progress  toward  accomplishing  key  event  and  schedule  interfaces 
•Helps  detect  disconnects  early,  and  thus  provide  sufficient  lead-time  and  a  planning 
tool  to  help  address  them 

•Is  able  to  be  traced  to  the  major  events  of  the  proposal  and,  upon  contract  award, 
trace  to  the  IMP/IMS 
•Is  kept  current 

DAPS  does  not  address  the  timeframe  of 
personnel  assignments,  just  feasibility  within 
the  program's  schedule 

3.3.6. C12:  In  a  SoS  program,  the  technical  planning  process  must  be  initiated  top-down  but 
iterated  within  individual  systems  until  a  consensus  approach  is  agreed  upon  and  resourced. 

3.4.2. Q9:  How  has  the  PM  addressed  intra-government  work  agreements,  i.e.,  formal  agreements, 
project  orders,  or  work  requests,  in  which  one  government  activity  agrees  to  perform  work  for 
another,  creating  a  supplier/customer  relationship? 

3.4.2. Q24:  How  have  teaming  agreements  been  documented,  defined,  and  communicated  among 
all  relevant  parties? 

•What  is  the  process  for  making  changes  to  agreements,  and  who  is  involved?  [3.4.2.Cla] 

4.1.6. C15:  The  PM  shall  work  with  the  users  to  document  performance  and  support  requirements 
in  performance  agreements  specifying  objective  outcomes,  measures,  resource  commitments, 
and  stakeholder  responsibilities.  The  military  Services  shall  document  sustainment  procedures 
that  ensure  integrated  combat  support. 

4.3.1.C30:  The  IBR  establishes  a  mutual  understanding  of  the  Performance  Measurement  Baseline 
(PMB)  and  provides  for  an  agreement  on  a  plan  of  action  to  evaluate  risks  inherent  in  the  PMB  and 


•Use  of  Integrated  Product  Teams.  When  properly  oriented  and  challenged,  the  multifunctional 
members  of  the  IPT  become  committed  to  program  success,  thereby  reducing  parochial  or 
functional  imbalances  that  could  otherwise  lead  to  future  instability.  [3.1.1.C1] 

Integrated  product  and  process  development  is  a  management  process  that  integrates  all 
activities  from  the  concept  of  a  new  defense  system  through  the  entire  life  cycle,  using 
multidisciplinary  teams,  called  Integrated  Product  Teams  (IPTs). 

"The  DoD  has  recognized  the  importance  of  integrated  product  teams  as  a  means  to  aid  the 
program  manager,  and  as  a  way  to  streamline  the  decision  process.  By  working  as  part  of  cross¬ 
functional  teams,  issues  can  be  identified  and  resolved  more  quickly,  and  stakeholder  involvement 
in  the  overall  success  of  the  program  can  be  maximized.  In  this  way  the  program 
manager  capitalizes  on  the  strengths  of  all  the  stakeholders  in  the  defense  acquisition  system." 


3.3.3.C1:  The  PMO  is  organized  to  execute  all  functions  in  preparation  for  Milestone  A  review  and 
TD  activities,  including  the  plan  for  formation  of  appropriate  Integrated  Product  Teams  (IPTs)  or 
their  equivalents.  Roles  and  responsibilities  are  clearly  defined  and  consistent  with  achieving  the 
TD  objectives. _ 
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2. 

Are  the  IPTs  staffed  by  highly  qualified  personnel,  as  in 

Qualified  staff 

2.3.1 

Sufficiency  of  Numbers  and  Qualifications 

Addresses  the  issue  of  quality  staff  in  general. 

2.3.1.C1:  There  is  an  established  program/process  in  the  program  management  office  (PMO)  that 

2. 

2.a(a)? 

but  does  not  differentiate  with  critical  areas 

provides  the  right  number  and  mix  of  qualified  personnel  to  successfully  execute  the  Technology 

2 

or  IPTs 

Development  (TD)  phase.  There  is  sufficient  flexibility  in  the  program  to  address  program 
shortfalls  through  the  use  of  Systems  Engineering  Technical  Assistance  (SETA)  contractor 
personnel. 

2.3.1.C2:  The  contractor  has  an  established  program  that  provides  the  right  number  and  mix  of 
qualified  personnel  to  successfully  execute  the  TD  phase.  Key  contractor  management  and 
technical  personnel,  including  the  program  manager,  chief  systems  engineer,  software  architect, 
and  functional  area  managers,  have  worked  successfully  on  projects  of  similar  complexity  and 
have  had  significant  work  experience  relevant  to  the  current  program  phase. 
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2. 

For  IPTs  addressing  strongly  coupled  objectives,  are 

Strongly  coupled 

Introduction 

DAPS  has  an  Overarching  Integrated  Product 

"Quick-look"  reviews  are  conducted  2  to  3  months  before  the  milestone,  using  the  same  form  and 

2. 

there  "super-IPTs"  for  resolving  conflicts  among  the 

3.2.1 

Statutory  and  Regulatory  Compliance  and 

Team  but  it  is  unclear  on  this  group's  purpose 

formats  as  a  full  assessment.  They  are  conducted  as  a  "for  record"  review  to  support  the  Defense 

3 

objectives? 

Guidance 

or  participation  in  the  program  during  all  life 

Acquisition  Board's  (DAB)  Integrated  Process/Product  Teams  (IPTs),  Overarching  Integrated 

1.1 

Concept  of  Operations 

cycles. 

Product  Teams  (OIPTs),  or  if  requested,  for  the  DAB." 

4.1.2 

Modular  Open  Systems  Approach 

3.2.1.Q5:  What  are  the  Service-specific  regulatory  requirements  for  the  program? 

DAPS  mentions  planning  for  resolving  issues 

•  Are  they  in  conflict  with  higher  level  (e.g.,  DoD)  regulations? 

throughout,  but  does  not  mention  "super 

•  What  are  the  impacts  of  any  conflict  to  the  program? 

IPTs"  perse. 

•How  have  these  conflicts  been  resolved?  If  not,  were  the  conflicts  addressed  at  the 

DAPS  also  recommends  MOSA  as  an 

01 PT" Developing  the  CONOPS  as  a  team  effort  helps  resolve  requirement  debates  and  facilitates 

approach  to  resolve  capability  coupling 

completeness  of  requirements." 

4.1.2.C2:  To  realize  open  systems  benefits,  programs  need  to  continually  measure  their  progress 
toward  achieving  MOSA-enabled  capabilities/objectives.  Percentage  of  key  interfaces  defined  by 
open  standards,  or  percentage  of  components/subsystems  modularized  (self-contained, 
decoupled,  and  encapsulated)  are  examples  of  open  systems-related  metrics. 
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2.3 

Establishment  of  necessary  resources  for  meeting 
objectives 

3.3 

Program  and  Project  Management 

46 

2. 

Have  decisions  about  the  use  of  one-shot,  incremental. 

Incremental  development 

3.3 

Program  and  Project  Management 

In  general,  DAPS  uses  the  focus  of  working 

"Program  management . . .  represents  the  integration  of  a  complex  system  of  differing  but  related 

3. 

or  evolutionary  development  been  validated  for 

3.3.1 

Program  Plan/Schedule 

together.  This  probably  implies  "acceptance" 

functional  disciplines  . . .  that  must  work  together  to  achieve  program  goals." 

1 

appropriateness  and  feasibility,  and  accepted  by  the  key 

but  is  not  explicitly  stated. 

3.3.1.C6:  During  program  planning,  the  government  created  a  top  level  program  schedule  or 

stakeholders? 

Roadmap  which  . . . 

•  Is  focused  on  and  conveys  the  "big  picture"  of  the  program  objectives,  capabilities  evolution, 
summary  schedule,  and  any  major  program  constraints 

3.3.1. Q36:  How  does  the  Government  Roadmap  Schedule  capture  the  plan  for  executing  the 
evolutionary  acquisition  (EA)  strategy,  with  either  a  spiral  or  incremental  development  process? 

3.1.1. C1:  The  program  manager  (PM)  is  developing  a  credible  Acquisition  Strategy  that  will  provide 
the  basis  for  meeting  program  objectives  and  therefore  will  be  an  aid  in  gaining  program 
acceptance  and  support.  The  credibility  of  the  Acquisition  Strategy  is  evaluated  on  five 
attributes: 

•  Realism 

•  Stability 

•  Resource  balance 

•  Flexibility 

•  Managed  risk 

3.1.1.Q22:  How  does  the  Acquisition  Strategy  identify  and  describe  the  approach  the  program  will 

use  to  achieve  full  capability:  an  evolutionary  approach  or  a  single-step 

approach? 

•What  is  the  rationale  for  choosing  the  approach? 

•If  an  evolutionary  approach  is  being  used,  how  is  Block  1  (the  initial  deployment 

capability)  described;  how  will  it  be  funded,  developed,  tested,  produced,  and  supported; 
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2. 

Have  system  definition,  development,  test,  and 

system  plans  feasibility 

3.1.1 

Acquisition  Strategy/Credibility 

What  is  meant  by  "validated"? 

Before  all  milestones:  3.1.1.C2:  The  Acquisition  Strategy  is  credible,  based  on  the  following  five 

3. 

evolution  plans,  budgets,  and  schedules  been  validated 

attributes:  realism,  stability,  resource  balance,  flexibility,  and  risk  management.  The  Acquisition 

2 

for  appropriateness  and  feasibility,  and  accepted  by  the 

In  general,  DAPS  uses  the  focus  of  working 

Strategy  provides  the  basis  for  meeting  program  objectives,  thereby  acting  as  an  aid  in  gaining 

key  stakeholders? 

together.  This  probably  implies  "acceptance" 

program  acceptance  and  support. 

but  is  not  explicitly  stated. 

3.1.1.C3:  The  Acquisition  Strategy  documents  the  ground  rules  and  assumptions  under  which  the 
program  was  started  and  upon  which  future  decisions  will  be  gauged.  It  becomes  more  definitive 

over  the  execution  of  the  program  in  describing  the  relationships  of  the  following  essential 

elements: 

•Requirements 
•Structure  and  Schedule 
•Acquisition  Approach 
•Risk  Management 
•Program  Management 
-Philosophy/Approach 
-Program  Resources 
-Information  Sharing  and  DoD  Oversight 
-Integrated  Digital  Environment  (IDE) 

-Defense  Contract  Management  Agency  (DCMA)  Support 
-Government  Property  in  the  Possession  of  Contractors  (GPPC) 

-Streamlining/Innovative  Acquisition 
-Simulation-Based  Acquisition  (SBA) 

-Software-Intensive  Programs 
•Design  Considerations 
-Technology  Transition 
•Support  Strategy 
•Business  Strategy  Competition 

48 

2. 

Is  there  a  valid  business  case  for  the  system,  relating  the 

Business  case 

1.1.1 

Mission  Description 

1.1.1.C1:  The  system's  mission  description  clearly  identifies  mission  need,  objectives,  and  general 

3. 

life  cycle  system  benefits  to  the  system  total  cost  of 

1.2 

Analysis  of  Alternatives 

capabilities.  Included  is  a  suitable  description  of  the  operational  (including  threat)  and  logistical 

3 

ownership? 

3.4.3 

Value  Engineering 

environments  envisioned  for  the  system.  Information  is  current. 

1.2.1. C1:  There  is  a  viable  Analysis  of  Alternatives  (AoA)  study  plan  that  defines  what  will  be 
accomplished  and  how  it  will  be  done.  Minimum  information  in  the  study  plan  will  include: 
background,  purpose,  scope,  acquisition  issues,  alternatives,  effectiveness  and  cost 
methodologies,  analytical  tools,  and  schedule  to  complete  the  AoA. 

1.2.1. C2:  The  Analysis  of  Materiel  Approaches  (AMA),  if  conducted,  provides  the  best  materiel 
approach  or  combination  of  approaches  to  provide  the  desired  capability  or  capabilities.  The  AMA 
determines  the  best  way(s)  to  use  materiel  approaches  to  provide  a  joint  capability.  Note: 
Generally,  the  AMA  will  not  consider  which  specific  "systems"  or  "system  components"  are  best. 
3.4.3.C2:  There  is  a  viable  Value  Engineering  system  plan,  within  the  goals  and  stipulations  of  the 
PMO's  VE  program,  to  effectively  guide  the  successful  development  of 

solutions  that  eliminate  or  modify  any  element  of  the  program  that  significantly 
contributes  to  the  overall  cost  without  adding  commensurate  value  to  overall 
system  performance  or  program  execution. 
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2.4 

Establishment  of  appropriate  source  selection. 

3.4 

Contracting 

Figure  3-3  Contracting  Management  Process 

contracting,  and  incentive  structures 

3.1 

Acquisition  Strategy  (Credibility) 

3.1.1.Q60:  What  contract  type(s)  are  identified  in  the  Acquisition  Strategy? 

•  Explain  why  the  contract  types  are  suitable,  including  considerations  of  risk  assessment  and 
reasonable  risk  sharing  by  the  government  and  the  contractor(s). 

•  How  does  the  strategy  explain  the  planned  contract  incentive  structure,  and  how  will  the 
contract  provide  incentives  for  the  contractor(s)  to  provide  the  contracted  product  or  services  at 
or  below  the  established  cost  objectives? 

-If  more  than  one  incentive  is  planned  for  a  contract,  what  is  the  explanation  of  how  the 
incentives  complement  each  other  and  do  not  interfere  with  one  another?  [3.1.1.C3] 

51 

2. 

Has  the  competitive  prototyping  option  been  addressed. 

competitive  prototyping 

4.1.1 

System  Assurance 

Competitive  prototyping  is  required,  but  it  is 

4.1.1.C3:  Pending  the  next  version  of  DoDI  5000.2,  "3. 5. 2.6.  A  list  of  known  or  probable  Critical 

4. 

and  the  decision  accepted  by  key  stakeholders? 

4.3.1 

Technical  Review  Planning 

unclear  what  needs  to  be  done  for  it. 

Program  Information  (CPI)  and  potential  countermeasures  such  as  Anti-Tamper  (AT)  in  the 

1 

preferred  system  concept  and  in  the  critical  technologies  and  competitive  prototypes  to  inform 
program  protection  (DoDD  5200.39,  Reference  (ai))  and  design  integration  during  in  the  TD 
phase." 

Pre-Milestone  B  (New  DoDI  5000.02) 

The  use  of  competitive  prototyping  is  required  by  the  Office  of  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology  and  Logistics  (OUSD(AT&L))  policy  through  the  Technology  Development 
phase  up  to  Milestone  B,  which  will  include  the  Preliminary  Design  Review. 

4.3.1.Q20:  How  did  competitive  prototyping  affect  the  SRR? 

(c)  2009  Systems  Engineering  Research  Center 
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2. 

4. 

2 

If  doing  competitive  prototyping,  have  adequate  plans 
and  preparations  been  made  for  exercising  and 
evaluating  the  prototypes,  and  for  sustaining  core 
competitive  teams  during  evaluation  and  down 
selection? 

competitive  prototyping 
plans 

4.3.1 

Technical  Review  Planning 

How  to  plan  for/use  competitive  prototyping 
in  a  contract  or  program  management 
perspective  is  not  really  addressed.  However, 
the  results  of  competitive  prototyping  are 
used/evaluated  at  the  major  reviews. 

Not  addressed  to  this  level;  however  it  addressed  in  many  areas. 

4.3.1. C13:  The  SRR  is  typically  held  well  in  advance  of  Milestone  B  to  allow  time  for  issue 
resolution  and  proper  executive  level  concurrence  on  process  and  results.  Technical  performance 
results  from  competitive  prototyping  should  factor  into  the  trade  space  for  system  requirements. 

4.3.1. C21:  The  TD  effort  should  mature  the  prototype  technologies  to  an  acceptable  level  of  risk  to 
proceed  to  EMDD  and  assessment  by  the  SRR.  The  results  of  the  TD  effort  should  be  reflected 

53 

2. 

4. 

3 

Is  the  status  of  the  contractor's  business  and  team 
"healthy,"  both  in  terms  of  business  indicators,  and 
within  the  industrial  base  for  the  program  area?  Is  the 
program  aligned  with  the  core  business  of  the  unit,  and 
staffed  adequately  and  appropriately? 

contractor  qualifications 

3.4 

Contracting 

The  wording  of  this  item  is  confusing.  It 
seems  to  combine  many  issues  into  one. 

How  is  "healthy"  defined/determined?  What 
does  "within  the  industrial  base  for  the 
program  area"  mean?  Does  this  mean 
financial  stability  of  the  company,  ability  to 
keep  qualified  staff,  and  whether  this 
program  is  totally  different  from  the 
company's  usual  business? 

DAPS  coveres  whether  a  contractor  is  qualified  in  many  areas,  for  example, 

3.4.1. Q16:  In  regard  to  sources,  to  what  extent  has  the  acquisition  team  accomplished  the 
following  contracting  functions: 

•Availability  of  qualified  sources? 

•  Determination  if  the  source  can  meet  the  need? 

•For  commercial  sources,  review  of  acquisition  histories,  conduct  of  market  research,  and 
preparation  of  source  lists  of  identified  sources? 

•Verification  that  a  Qualified  Bidders  List,  Qualified  Manufacturers  List,  or  Qualified  Parts  List 
(QBL/QML/QPL)  applies  to  the  procurement? 

•Determination  from  market  research  whether  unlisted  firms  or  products  may  be  able  to  meet 
the  minimum  functional  need?  [3.4.1.C2b]3.4.1.Q56: 

3.4.1. Q56:  To  what  extent  were  past  performance,  technical  and  non-price  factors  addressed 
applied  by  the  acquisition  team? 

•How  was  the  latest  performance  information  in  the  Service's  contractor  performance  assessment 
reporting  system  used? 

3.4.2. Cla:  The  PMO  and  contractor  have  adequately  addressed  pre-award  activities  during  the 
preparation  of  the  solicitation. 

-  Analysis  of  the  Industrial  Base  -  PM  has  determined  the  capabilities  of  the  national  technology  ar 

3.4.2. Q5:  What  are  the  results  of  the  PM's  analysis  of  product  and  technology  areas 
critical  to  meeting  program  needs? 

•How  does  the  Acquisition  Strategy  identify  the  potential  industry  sources  to  supply  these  needs? 
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2. 

4. 

4 

Has  the  acquiring  organization  successfully  completed 
projects  similar  to  this  one  in 
the  past? 

Acquisitioner's  experience 

2.3 

Introduction 

Staffing  Level 

Program  Support  Team 

Structure:  The  Program  Support  Team  PST  is  composed  of  a  team  leader  from  SSE/AS  and  core 
subject  matter  expert  members  from  OSD  staff  (AT&L,  CAIG,  DPAP,  Networks  and  Information 
Integration  (Nil),  and  Director,  Operational  Test  and  Evaluation  (DOT&E)).  Additional  subject 
matter  experts  may  be  recruited  from  the  Services,  DoD  agencies.  Federally  Funded  Research  and 
Development  Centers  (FFRDCs),  and  academia  based  on  specific  assessment  needs  matched  with 
individual  expertise. 

-"The  key  to  applying  the  assessment  process  successfully  is  to  select  a  highly  qualified, 
experienced  team  leader,  and  populate  the  team  with  experienced  senior  individuals.  Collectively, 
the  assessment  team  should  bring  expertise,  experience,  and  knowledge  in  all  areas  that  the 
assessment  will  address." 

2.3:  Perspective:  Staffing  is  key  to  the  ability  of  any  PMO  to  execute  its  responsibilities.  Composed 
of  civilian,  military,  matrix  support,  and  Systems  Engineering  Technical  Assistance  (SETA)  (aka 
onsite  support  contractors),  the  staff  is  professional,  agile,  and  motivated.  It  consistently  makes 
smart  business  decisions,  acts  in  an  ethical  manner,  and  delivers  timely  and  affordable  capabilities 
2.3.1.Q4:  What  is  the  experience  level  of  each  of  the  existing  or  planned  key  technical  personnel? 
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2. 

4. 

5 

Has  the  candidate  performing  organization  successfully 
completed  projects  similar  to  this  one  in  the  past? 

contractor  qualifications/ 
experience 

2.3.1 

Sufficiency  of  Numbers  and  Qualifications 

2.3.1.C4:  The  contractor  has  an  established  program  that  provides  the  right  number  and  mix  of 
qualified  personnel  to  successfully  execute  the  SDD  phase.  Key  contractor  management  and 
technical  personnel,  including  the  program  manager,  chief  systems  engineer,  software  architect, 
and  functional  area  managers,  have  worked  successfully  on  projects  of  similar  complexity  and 
have  had  significant  work  experience  relevant  to  the  current  program  phase.  The  contractor's 
policy  and  actual  practice  on  workforce  assignments  reflect  a  commitment  to  a  stable  workforce 

56 

2. 

4. 

6 

Is  the  program  governance  process,  and  in  particular  the 
system  engineering  plan,  well  articulated  and 
compatible  with  the  goals  of  the  program? 

3.3.1 

Program  Plan/Schedule 

The  term  "Program  governance  process"  is 
confusing-not  sure  how  this  relates  to  the 
system  engineering  plan. 

Systems  Engineering  Plan  (SEP)  -  The  IMP  and  IMS  supports  the  sound  technical  approach 
documented  in  the  SEP.  The  IMP  demonstrates  the  contractual  commitment  to  the  elements  of 
major  technical  reviews  and  their  entry  and  exit  (success)  criteria.  The  SEP  and  IMS  demonstrate 
that  Cost,  Schedule  and  Performance  are  inter-related  within  the  program.  Note:  Because  the 
basic  tasks  within  the  IMS  track  the  systematic  flow  of  the  engineering  process,  there  should  be  a 
relationship  between  the  SEP  and  the  IMS.  These  processes,  tools,  and  documents  should  be 
understood,  linked,  and  tailored  for  an  individual  program's  execution  needs  and  management 
reporting  requirements 

3.3.1.Q32:  How  does  the  program  ensure  that  all  key  strategies  and  top-level  plans  remain 
consistent  and  aligned  (i.e.,  coordinated)  with  the  IMP/IMS? 

•Are  the  type  and  number  of  technical  reviews  correct  in  each  appropriate  plan? 

•Does  the  IMS  capture  both  the  government  SEP  and  the  prime  contractor's  SEMP/SEP  activities, 
events,  and  milestones? 

•  Are  the  scheduled  interfaces  w  FoS/SoS  correctly  captured  in  the  IMS,  SEP,  TEMP,  and  other 
related  plans? 

•Did  the  plans  adequately  address  or  reference  all  key  processes  (e.g..  Requirements,  Risk 
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2.5 

Establishment  of  necessary  personnel  competencies 

2.3.1 

Sufficiency  of  Numbers  and  Qualifications 

Have  the  appropriate  technical  and  programmatic  competencies  been  involved  in  the  CARD-like 

2.2.1 

Program  Funding  and  Allocation 

document  development,  and  have  the  proper  subject  matter  experts  been  involved  in  its  review? 
(CARD-Cost  Analysis  Requirements  Description 
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2. 

Does  the  government  have  access  over  the  life  of  the 

qualified  Government  stafl 

2.3.1 

Sufficiency  of  Numbers  and  Qualifications 

2.3.1. Cl:  There  is  an  established  program/process  in  the  program  management  office  (PMO)  that 

5. 

program  to  the  talent  required  to  manage  the  program? 

provides  the  right  number  and  mix  of  qualified  personnel  to  successfully  execute  the  Technology 

1 

Does  it  have  a  strategy  over  the  life  of  the  program  for 

Development  (TD)  phase.  There  is  sufficient  flexibility  in  the  program  to  address  program 

using  the  best  people  available  in  the  government,  the 

shortfalls  through  the  use  of  Systems  Engineering  Technical  Assistance  (SETA)  contractor 

FFRDCs,  and  the  professional  service  industry? 

personnel. 

2.3.1.Q6:  Are  the  personnel  (e.g.,  program  management,  contracting,  oversight)  trained  to  the 
appropriate  levels  in  accordance  with  their  acquisition  career  assignments? 

•  Are  government  PMO  personnel  in  acquisition-critical  positions  trained  to  the  appropriate 
certification  levels  in  accordance  with  their  acquisition  career  assignments?  [2.3.1.C1] 
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2. 

At  Milestone  B,  have  sufficiently  talented  and 

qualified  program 

2.3.1 

Sufficiency  of  Numbers  and  Qualifications  (pre- 

The  two  questions  should  be  separated. 

2.3.1.C3:  The  PMO  staff  is  the  right  mix  of  qualified  personnel  to  successfully  execute  the  System 

5. 

experienced  program  and  systems  engineering 

management,  system 

milestone  B) 

Empowerment  is  different  from  qualified 

Development  and  Demonstration  (SDD)  phase.  Workforce  management  and  training  programs 

2 

managers  been  identified?  Have  they  been  empowered 

engineering; 

3.3.6 

Management  of  Dependencies  and  External 

persons. 

receive  the  highest  priority  in  resources  to  ensure  a  qualified  workforce  to  complete  the  SDD 

to  tailor  processes  and  to  enforce  development  stability 

empowered  staff 

Interfaces 

phase  and  transition  to  production.  There  is  sufficient  flexibility  in  the  program  to  address 

from  Milestone  B  through  IOC? 

program  shortfalls  through  the  use  of  SETA  contractor  personnel.  Policies  and  standards  are  in 
place  to  ensure  the  thorough  and  continual  training  of  the  workforce  and  to  evaluate  worker 
performance. 

2.3.1.C4:  The  contractor  has  an  established  program  that  provides  the  right  number  and  mix  of 
qualified  personnel  to  successfully  execute  the  SDD  phase.  Key  contractor  management  and 
technical  personnel,  including  the  program  manager,  chief  systems  engineer,  software  architect, 
and  functional  area  managers,  have  worked  successfully  on  projects  of  similar  complexity  and 
have  had  significant  work  experience  relevant  to  the  current  program  phase.  The  contractor's 
policy  and  actual  practice  on  workforce  assignments  reflect  a  commitment  to  a  stable  workforce  tf 
3.3.6.Q11:  •  Is  the  SE&I  lead  empowered  to  integrate  the  programs  within  the  SoS,  and 
reallocate  resources  (e.g.  funding  and  manpower)  within  the  SoS  from  the  "fast  movers" 
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2. 

Has  the  government  attempted  to  align  the  duration  of 

Length  of  assignment  is  not  addressed  in 

Length  of  assignment  is  not  addressed 

5. 

the  program  manager's  assignment  with  key 

DAPS  other  than  indirectly  by  "realism" 

3 

deliverables  and  milestones  in  the  program? 

guidelines 
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2. 

Is  the  quantity  of  systems  engineering  personnel 

2.3.1 

Sufficiency  of  Numbers  and  Qualifications  (pre- 

2.3.1.Q14:  How  has  the  contractor  committed  to  having  a  quality  workforce  throughout  the  TD 

5. 

assigned,  their  skill  and  seniority  mix,  and  the  time 

milestone  B) 

phase?  [2.3.1.C2] 

4 

phasing  of  their  application  throughout  the  program  life 

Also,  see  above. 
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3  Technology  maturing,  architecting 

65 

66 

3.1 

COTS/NDI  evaluation,  selection,  validation  for  maturity 
&  compatibility 

3.1.1 

Acquisition  Strategy 

67 

3. 

Have  COTS/NDI/Services  opportunities  been  evaluated 

COTS/NDI  evaluation 

3.1.1 

Acquisition  Strategy  (Credibility) 

3.1.1.Q52:  How  does  the  Acquisition  Strategy  consider  both  commercial  and  NDI  sources  as  the 

1. 

prior  to  baselining  requirements? 

primary  source  of  supply?  What  is  the  role  of  market  research  in  determining  the  availability  and 

1 

suitability  of  commercial  and  NDIs,  and  to  what  extent  do  the  interfaces  for  these  items  have 
broad  market  acceptance,  standards-organization  support,  and  stability? 

•What  is  the  role  of  commercial  off-the-shelf  (COTS)  and  NDI  sources  of  supply  to  provide  for  the 
most  cost-effective  system  throughout  the  system's  life  cycle? 

•  How  does  the  PM  work  with  the  user  to  define  and  modify,  as  necessary,  requirements  to 
facilitate  the  use  of  COTS  items  and  NDIs?  [3.1.1.C3] 

68 

3. 

Have  COTS/NDI/Services  scalability,  compatibility. 

3.1.1 

Acquisition  Strategy  (Credibility) 

Suitability  is  addressed  which  should  include 

1. 

quality  of  service,  and  life  cycle  support  risks  been 

these  items,  but  all  of  them  are  not  called  out 

2 

thoroughly  addressed? 

separately. 

69 

3. 

Has  a  COTS/NDI/Services  life  cycle  refresh  strategy  been 

COTS  refresh 

3.1.2 

Acquisition  Strategy  (Acceptability) 

3.1.2.C1:  Before  development  of  a  program  Acquisition  Strategy  in  the  Technology  Development 

1. 

developed  and  validated? 

(TD)  phase,  a  Technology  Development  Strategy  (TDS)  is  formulated  during  the  Concept 

3 

Refinement  (CR)  phase  and  approved  by  the  MDA  at  Milestone  A.  The  TDS  contains  the  research 
and  development  strategy  to  be  implemented— particularly  in  the  TD  phase— and  the  rationale 
for  the  planned  acquisition  approach  to  achieve  full  capability. 

3.1.2. Q12:  How  is  technology  obsolescence  factored  into  the  TDS? 

•  Does  the  strategy  include  a  process  to  determine  when  technology-refresh  actions  should  be 
performed?  If  not,  why  not?  [3.1.2.C1] 

3.1.2. Q29:  How  does  the  Acquisition  Strategy  describe  the  program's  approach  for  applying 
Modular  Open  Systems  Approach  (MOSA),  as  characterized  by  the  following  attributes? 

...•Commercial-off-the-shelf  (COTS)  technology  refreshment  plans 

70 

71 

3.2 

Life-cycle  architecture  definition  &  validation 

4.1.3 

Architecture 

(c)  2009  Systems  Engineering  Research  Center 
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3. 

Has  the  system  been  partitioned  to  define  segments 

system  partitioning 

4.2.1 

Analysis  and  Decomposition 

The  wording  of  this  question  is  more  software 

4.2.1.C4:  The  program  manager  (PM)  or  contractor  has  an  effective  systems  engineering  (SE) 

2. 

that  can  be  independently  developed  and  tested  to  the 

oriented;  the  DAPS  is  more  system-oriented 

process  in  place  to  perform  functional  analysis  and  the  allocation  of  functional  requirements  for 

1 

greatest  degree  possible? 

the  TD  phase.  This  includes  the  traceability  and  verification  of  requirements  across  the  entire 
system. 

•  The  SE  process  is  effective  in  defining  system  requirements,  functionality,  and  allocated  physical 
architecture. 

•Technology  maturity  requirements  are  appropriately  scoped  for  demonstration  during  the  TD 
phase. 

•Analyses  provide  a  clear,  detailed  description  of  the  technical  approach  resulting  from  functional 
analysis  and  allocation. 

•  The  SE  process  uses  rigorous  and  disciplined  definitions  of  interfaces,  and  defines  the  key 
interfaces  that  require  test  verification  within  the  system. 

•  The  SE  process  partitions  the  system  into  self-contained,  groupings  of  interchangeable  and 
adaptable  modules.  The  process  enables  identification  of  key  test  and  evaluation  (T&E) 

73 

3. 

By  Milestone  A,  is  there  a  plan  to  have  internal  and 

information  exchange 

1.3.2 

Key  Performance  parameters  and  Key  System 

Pre  milestone  B:  1.3.2.C8:  Programs  will  use  standardized  architectural  products  and  conventions. 

2. 

external  information  exchange  protocols  established  for 

Attributes 

data  formats,  and  open  interface  standards  and  protocols  to  enable  interoperability. 

2 

the  whole  system  and  its  segments  by  Milestone  B? 

4.2.1 

Analysis  and  Decomposition 

Pre  milestone  A:  4.2.1.C1:*  The  system  architecture  is  well  defined  and  documented,  and  is  in 
accordance  with  all  applicable  standards,  protocols  and  data  interchange  definitions  as  defined  by 
key  interface  descriptions. 

74 

3. 

Does  the  project  have  adequate  processes  in  place  to 

4.6.1 

Test  and  Evaluation  Plan 

4.6.1.C1:  The  program  manager  (PM)  shall  develop  a  robust  integrated  Test  and  Evaluation 

2. 

define  the  verification,  test  &  validation,  and  acceptance 

Strategy  (TES)  for  all  phases  of  the  program,  describing  developmental  test  and  evaluation  (DT&E), 

3 

of  systems  and  system  elements  at  all  phases  of 

operational  test  and  evaluation  (OT&E),  and  live-fire  test  and  evaluation  (LFT&E).  Without 

definition  and  development? 

compromising  rigor,  the  program  is  required  to  integrate  modeling  and  simulation  (M&S)  activities 
into  the  strategy.  The  TES  should  be  consistent  with  and  complementary  to  the  Systems 

75 

3. 

Is  there  a  clear,  consistent,  and  traceable  relationship 

1.3.1 

Capatilities-Reasonableness,  Stability,  and 

In  DAPS,  it  is  not  explicitly  required  to  have 

1.3.1.C10:  Compatibility  with  other  interfacing  systems  is  maintained  and  verified  through  system- 

2. 

between  system  requirements  and  architectural 

Testability 

traceability  between  reqs  and  architecture 

level  testing  as  defined  in  interface  specifications,  through  the  development/design  process,  and 

4 

elements?  Have  potential  off-nominal  architecture- 

4.1.3 

Design  Considerations/Architecture 

is  traceable  to  the  architecture  of  the  system.  Interface  specifications  are  under  formal 

breakers  been  addressed? 

4.2.1 

Analysis  and  Decomposition 

configuration  control. 

4.1.3.Q3:  How  is  a  change  within  the  technical  system  descriptions  ensured  for  traceability  of 
impact  across  the  system?  [4.1.3.C3] 

4.2.1. C4:  The  program  manager  (PM)  or  contractor  has  an  effective  systems  engineering  (SE) 
process  in  place  to  perform  functional  analysis  and  the  allocation  of  functional  requirements  for 
the  TD  phase.  This  includes  the  traceability  and  verification  of  requirements  across  the  entire 
system. 

•  The  SE  process  is  effective  in  defining  system  requirements,  functionality,  and  allocated  physical 
architecture. 

4.2.1. C10:  The  PM  or  contractor  has  an  effective  SE  process  in  place  to  perform  functional  analysis 
and  the  allocation  of  functional  requirements  for  the  SDD  phase.  This  includes  the  traceability  and 
verification  of  requirements  across  the  entire  system. 

•The  SE  process  is  effective  in  defining  system  requirements,  functionality,  and  allocated 

76 

3. 

Does  the  architecture  adequately  reconcile  functional 

Architecture 

4.1.3 

Architecture 

Unclear  of  the  terminology  used  in  this 

DAPS  not  written  this  way--think  the  following  covers  it: 

2. 

hardware  "part-of"  hierarchies  with  layered  software 

decomposition 

question 

4.1.3.C1:  The  system  architecture  and  subsystem  architecture,  including  computer  system  and 

5 

"served-by"  hierarchies? 

support  architectures,  is  defined  using  standardized  methods,  such  as  the  Department  of  Defense 
Architecture  Framework  (DoDAF),  and  widely  accepted  tools  sets,  such  as  those  that  employ  the 
Unified  Modeling  Language  (UML),  which  meets  the  system  requirements,  including  open-system 
requirements  and  benefits.  Ease  of  change,  growth,  upgrade,  and  lifecycle  support  is  facilitated 
with  this  architecture. 

4.1.3. C2:  The  technical  system  architecture  descriptions  should  use  mandated  Operational  View 
(OV),  System  View  (SV),  and  Technical  View  (TV)  products  as  described  in  the  DoDAF,  and  should 
be  integral  to  the  system  design.  There  should  be  System  Description  Documents  (SDDs)  and 
System  Capability  Specifications  (SCSs)  that  address  those  for  the  system  and  major  subsystems. 

4.1.3. C3:  There  should  be  a  disciplined  process  to  ensure  that  the  technical  system  descriptions 

are  integrated  such  that  changes  to  any  one  that  affects  others  is  identified  and  tracked  to  conclus 

77 

3. 

Has  a  Work  Breakdown  Structure  (WBS)  been  developed 

WBS 

3.3.2 

Work  Breakdown  Structure 

3.3.2.Q2:  For  a  joint  and/or  System  of  Systems  (SoS)  program,  does  the  WBS  identify  and  describe 

3. 

with  the  active  participation  of  all  relevant  stakeholders. 

the  "parent-child"  type  relationship?  Note:  Understanding  the  parent-child  type  relationship  of 

6 

which  accurately  reflects  both  the  hardware  and  the 

various  related  programs  and  contracts  and  their  impact  on  the  WBS  is  important  in  the  ever- 

software  product  structure? 

increasing  integrated  and  joint  program  environment.  Often,  individually  base-lined  programs  and 
their  various  prime  or  GFE  elements  are  actually  part  of  a  SoS  approach.  The  overall  parent 
program  -  the  SoS  or  joint  program,  needs  to  be  identified  with  the  various  child  programs.  Each 
child  program  would  develop  a  stand-alone  WBS  structure  [3.3.2.C1] 

3.3.2.C9:  The  Contract  WBS  (CWBS)  is  the  complete  WBS  as  included  in  the  DoD-approved  PWBS 
extended  to  the  agreed-to  contract  reporting  level  and  any  discretionary  extensions  to  lower 
levels  for  reporting  or  other  purposes.  It  adequately  defines  the  lower  level  components  of  what  is 
to  be  procured  and  includes  all  the  product  elements  (hardware,  software,  data,  or  services), 
which  are  defined  by  the  contractor. 

78 
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3.1.1. C3:  The  Acquisition  Strategy  documents  the  ground  rules  and  assumptions  under  which  the 
program  was  started  and  upon  which  future  decisions  will  be  gauged.  It  becomes  more  definitive 
over  the  execution  of  the  program  in  describing  the  relationships  of  the  following  essential 
elements: 

-Simulation-Based  Acquisition  (SBA)  -  Acquisition  strategy  should  address  SBA,  the  robust  and 
interactive  use  of  modeling  and  simulation  (M&S)  throughout  the  product  life  cycle. 

3.1.1. Q33:  How  does  the  Acquisition  Strategy  describe  the  PM's  use  of  Simulation-Based 
Acquisition  (SBA)  throughout  the  product  life  cycle?  Note:  The  PM  should  use  SBA  and  M&S 
during  system  design,  system  T&E,  and  system  modification  and  upgrade.  In  collaboration  with 
industry  and  operational  users,  PMs  should  integrate  SBA/M&S  into  program  planning  activities; 
should  plan  for  life  cycle  application,  support,  documentation,  and  reuse  of  models  and 
simulations;  and  should  integrate  SBA/M&S  across  the  functional  disciplines  [3.1.1.C3] 

•Design  Readiness  Review.  Knowledge  should  indicate  that  the  product  can  be  built  consistent 
with  cost,  schedule,  and  performance  parameters.  This  means  design  stability 

and  the  expectation  of  developing  one  or  more  workable  prototypes  or  engineering 
development  models. 

4.2.3. C4:  The  results  of  a  demonstration/validation  of  new  or  advanced  technologies  quantify  risk 
elements,  and  support  the  design  strategy.  A  risk  mitigation  plan  is  initially  developed  to  address 
the  attendant  risks,  including  adequate  resources  and  schedule  to  accomplish  planned  mitigation 
activities. 

4.2.3. Q7:  What  is  the  plan  for  the  demonstration  and  validation  of  the  proposed  technologies  and 
the  quantifiable  risks  that  remain  to  mature  the  technologies  for  system  development  and 
integration? 

•What  are  the  risk  mitigation  plan  and  the  resources  required  to  validate  (i.e.,  verification  testing, 
modeling  and  simulation,  etc)?  [4.2.3. C4] 

Pre  milestone  B-- 

4.2.3. C7:  The  SE  process  manages  technology  maturation  within  the  context  of  the  documented 
Technology  Development  Strategy  (TDS),  and  manages  the  associated  risk. 

4.2.3. C8:  Fiscal  Year  2006,  Public  Law  109-163,  Section  801  requires  that  the  Under  Secretary  of 
Defense  for  Acquisition,  Technology  and  Logistics  (USD(AT&L))  certify,  before  Milestone  B,  that 
"the  technology  in  the  program  has  been  demonstrated  in  a  relevant  environment."  This  wording  ( 
immature  critical  technology,  a  more  mature  alternative  technology  has  been  identified  in 

order  to  reduce  the  program  risk  if  the  immature  technology  does  not  mature  as  planned. 

This  is  described  in  the  Critical  Technology  Element  (CTE)  maturation  plan,  which 
explains  in  detail  how  the  required  TRL  will  be  reached  prior  to  the  next  milestone  decision 
date  or  relevant  decision  point.  This  plan  includes  the  identification  of  adequate  resources 
and  schedule  to  accomplish  planned  mitigation  activities. 

The  following  knowledge  points  coincide  with  decisions  along  the  acquisition  framework: 

•Program  Initiation.  Knowledge  should  indicate  a  match  between  the  needed  capability  and 
available  resources  before  a  program  starts.  In  this  sense,  resources  is  defined  broadly,  to  include 
technology,  time,  and  funding. 

•Design  Readiness  Review.  Knowledge  should  indicate  that  the  product  can  be  built  consistent 
with  cost,  schedule,  and  performance  parameters. 

The  SRR  is  intended  to  confirm  that  the  user's  requirements  have  been  translated  into  system- 
specific  technological  requirements,  that  critical  technologies  are  identified,  required  technology 
demonstrations  are  planned,  risks  are  well  understood,  and  mitigation  plans  are  in  place  [3.2. 2. C7] 

1.1.1. Q17:  Is  there  traceability  among  the  CONOPS,  the  capabilities/requirements  generation 
process,  and  system  performance  parameters  to  validate  the  end  product  through  test  and 
evaluation  (T&E)?  [1.1.1.C7] 

Capabilities  Development  Document  (CDD).  The  CDD  replaced  the  Operational  Requirements 
Document. 

The  CDD  will  be  validated  and  approved  before  Milestone  B.  The  CDD  will  be  validated  and 
approved  prior  to  program  initiation  for  shipbuilding  programs.  The  CDD  provides  the  operational 
performance  attributes  necessary  for  the  acquisition  community  to  design  a  proposed  system(s) 
and  establish  a  program  baseline.  It  identifies  the  performance  attributes,  including  Key 
Performance  Parameters  (KPPs),  that  guide  the  development  and  demonstration  of  the  proposed 
system. 

Capability  Production  Document  (CPD).  The  CPD  is  the  sponsor's  primary  means  of  providing 
authoritative,  testable  capabilities  for  the  Production  and  Deployment  phase.  A  CPD  is  finalized 
after  the  design  readiness  review  and  must  be  validated  and  approved  before  the  Milestone  C 
acquisition  decision.The  CPD  refines  the  threshold  and  objective 
values  for  the  performance  attributes  and  KPPs  that  are  validated  in  the  CDD  for  the 
production  phase.  The  refinement  of  performance  attributes  and  KPPs  is  the  most 
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3. 

Has  the  claimed  degree  of  reuse  been  validated? 

3.3.4 

Management  Methods,  Metrics,  and  Techniques 

3.3.4.3.C8:  The  Government  Program  Office  should  initially  approve  the  program  metrics  and  then 

3. 

4.5.1 

Software  Development  Plan 

periodically,  e.g.,  monthly,  the  metrics  should  be  reported  and  reviewed.  These  metrics  should 

4 

include  many,  if  not  all  of  the  following:  Development  status  S  curves;  Processor  throughput 
utilization;  Processor  memory  utilization;  Input/output  utilization;  Software  Engineering  Staffing; 
Software  Work  Packages  Summary;  Schedule  Performance  Index;  Cost  performance  Index; 
Problem/Deficiencies  /Discrepancies  Status;  Requirements  Stability;  Software  Size;  Software 

Reuse  Status  (planned  versus  'actuals');  Reliability  Growth  Curve;  Logistics  Footprint  Reduction; 
Planned  Operational  Effectiveness;  Product  Availability  Predictions;  O&S  Cost  Projections; 
Development  Test  entrance  criteria  and  status;  DAES  Reporting  (For  MDAPS);  Milestone  B  and  C 
entrance  criteria. 

4.5.1.C17:  Reuse  of  software,  from  existing  systems  or  prior  development  efforts,  has  been 
analyzed  for  complexity  and  suitability  to  meet  required  functionality,  in  accordance  with 
accepted  software  engineering  standards.  Pre-Milestone  C,  this  analysis 
has  resulted  in  documented  re-use  in  line  with  plans. 

84 

85 

3.4 

Validated  system  engineering,  development, 

3.3.1 

Program  Plan/Schedule 

•  Integrated  Baseline  Review  (IBR)  -  The  IMS  facilitates  the  conduct  of  a  successful  IBR,  in  which  it 

manufacturing,  operations  &  maintenance  budgets 

is  verified  there  is  sound  basis  for  cost  and  schedule  execution  of  program  objectives,  program 

and  schedules 

risks  are  addressed  and  the  contractor  has  performance  plans  and  underlying  management 
control  systems  to  assess  the  realism  of  the  performance  measurement  baseline  providing  the 

86 

3. 

Are  the  major  known  cost  and  schedule  drivers  and  risks 

2.1 

Resources/Program  Schedule  Overview  (TIER  1) 

Perspective:  Experienced  program  personnel  provide  data  regarding  critical  and  high-risk  efforts 

4. 

explicitly  identified,  and  is  there  a  plan  to  track  and 

Program  Funding  and  Allocation 

and  identify  as  realistically  as  possible  the  expected  schedule,  which  the  program  management 

1 

reduce  uncertainty? 

2.2.1 

Acquisition  Strategy 

office  then  compares  with  the  top-level  defense  program  schedule  template  to  determine  the 

3.1.2 

Entrance  and  Exit  Criteria 

actual  schedule  risk  and  to  identify  all  schedule  drivers.  With  this  approach,  the  probability  of 

3.2.2 

overrunning  a  program  schedule  can  be  estimated  by  determining  how  much  risk  exists  and 
where  it  is  greatest.  This  approach  enables  program  managers  (PMs)  to  estimate  early  and 
continuously  in  the  program  the  possibility  of  a  significant  likelihood  of  overrunning  the  program 
schedule  by  determining  how  much  and  where  the  risk  to  successful  schedule  completion  is 
greatest. 

2.2.1. Q13:  Is  the  program,  as  captured  in  the  CARD-like  document,  executable? 

•  Does  the  CARD-like  document  capture  the  key  program  cost  drivers,  development  costs  (all 
aspects  of  hardware,  human  integration,  and  software),  production  costs,  and  operation  and 
support  costs? 

3.1.2. C5:  The  Acquisition  Strategy  and  specific  acquisition  approaches  are  consistent  with  operatio 

•  Program  risks  are  identified  and  documented,  and  progress  is  tracked  via  established  metrics  tha 

3.2.2. Q48:  Did  the  following  result  from  the  SRR? 

•  A  comprehensive  risk  assessment  for  the  SDD  phase 

87 

3. 

Have  the  cost  confidence  levels  been  developed  and 

2.2.2 

Continuity  and  Stability 

Assuming  acceptance  is  implied  because  it 

pre  all  milestones: 

4. 

accepted  by  the  key  system  stakeholders? 

was  created  with  the 

2.2.2.C1:  Flow  of  funding  is  stable  and  steady  throughout  the  phases  of  the  system's  acquisition 

2 

contractors/stakeholders  involved 

life  cycle.  The  program  manager  (PM)  and  contractor  plan  for  perturbations  in  the  budget,  both 
from  within  and  outside  their  spectrum  of  control.  Accordingly,  the  PM  has  taken  the  following 
minimal  steps  to  achieve  greater  control  over  maintaining  a  stable  budget:  obtaining  a  high- 
confidence  cost  estimate  that  is  well  documented  to  firmly  support  budget  requests;  ensuring 

user  advocacy  for  the  program;  ensuring  that  funding  for  the  execution  year(s)  is  consistent  with 

the  contractor's  ability  to  expend  the  funding  according  to  the  current  program  schedule;  and 
developing  a  range  of  independent  estimates  at  completion  from  earned  value  data  and  analysis 
of  the  integrated  master  schedule.  Compare  the  results  with  the  contractor's  projected  final  costs 
to  assess  realism  and  to  form  the  basis  for  adjusting  the  program  budget. 
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3. 

Is  there  a  top-to-bottom  plan  for  how  the  total  system 

1.3.1 

Reasonableness,  Stability,  and  Testability 

The  CPD  captures  the  information  necessary  to  support  production,  testing,  and  deployment  of  an 

4. 

will  be  integrated  and  tested?  Does  it  adequately 

4.6.1 

Test  and  Evaluation  Plan 

affordable  and  supportable  system  within  an  Acquisition  Strategy.  The  CPD  refines  the  threshold 

3 

consider  integration  facilities  development  and  earlier 

and  objective  values  for  the  performance  attributes  and  KPPs  that  are  validated  in  the  CDD  for  the 

integration  testing? 

production  phase. 

4.6.1. C1:  The  program  manager  (PM)  shall  develop  a  robust  integrated  Test  and  Evaluation 
Strategy  (TES)  for  all  phases  of  the  program,  describing  developmental  test  and  evaluation  (DT&E), 
operational  test  and  evaluation  (OT&E),  and  live-fire  test  and  evaluation  (LFT&E).  Without 
compromising  rigor,  the  program  is  required  to  integrate  modeling  and  simulation  (M&S)  activities 
into  the  strategy.  The  TES  should  be  consistent  with  and  complementary  to  the  Systems 
Engineering  Plan  (SEP). 

4.6.1. C3:  The  system  integration,  test  and  evaluation  process  is  defined  in  the  Technology 
Development  Strategy  (TDS)  and  includes  analysis,  reviews,  inspections,  demonstrations,  testing, 
and  M&S  to  evaluate  the  requirements  baseline  and  the  system's  progress  during  development  to 
meet  the  critical  technical  parameters  (CTPs).  The  TES  describes  an 

iterative  process  by  which  allocated  specifications  and  CTPs  are  met  by  lower-level 
components,  assemblies,  subsystems  and  then  at  the  system  level.  Requirements 
are  traceable  to  specific  test  and  evaluation  events. 

4.6.1. C7:  Integration  test  facilities  that  allow  demonstration  of  hardware  and  software 

(c)  2009  Systems  Engineering  Research  Center 
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3. 

If  time-boxing  or  time-determined  development  is  used 

4.2.2 

Management  of  Requirements 

DAPS  does  not  address  "prioritized" 

4.2.2.C10:  The  evolutionary  Acquisition  Strategy  (AS)  utilizes  a  management  system  that 

4. 

to  stabilize  schedules,  have  features  been  prioritized  and 

1.3 

Capabilities 

requirements  but  requires  a  fexible  process 

continually  defines  the  requirements  and  development  activities  to  support  the  evolving  needs; 

4 

the  system  architected  for  ease  of  adding  or  dropping 

2.2 

Constraints  and  Dependencies 

that  supports  change  when  needed.  It  also 

adequately  addresses  the  various  concerns  of  users,  developers,  and  managers;  and  mitigates  the 

borderline  features? 

recommends  a  modular  and  open 

risks  associated  with  these  issues.  The  basic  system  architecture  is  designed  to  accommodate 

architecture  to  facilitate  change. 

change.  Techniques  such  as  open  systems  design,  functional  partitioning  and  modular  design  have 
been  addressed  by  the  PM  to  achieve  a  flexible  system  that  can  be  easily  and  affordably  modified. 

Time-boxing  or  time-determined 

1.3:  New  capabilities  are  defined  within  the  "art  of  the  possible"  and  grounded  within  real-world 

development  is  not  mentioned,  but  it 

constraints  of  time,  technology,  and  affordability. 

mentions  accomodating  constraints  of  time. 

1.3.1.C1:  Milestone  A  review . . .  The  ICD  clearly  states  required  capabilities  in  broad  and  time- 

It  also  recommends  a  time  reserve. 

phased  operational  goals. 

2.1.2.C1. .  .The  end  result  is  a  program  schedule  that  has  inherent  flexibility  to  accommodate  the 
competing  demands  of  time  and  resources  while  ensuring  the  best  capability  to  the  warfighter. 
Note:  Constraints  are  effectively  global  requirements,  such  as  limited  development  resources  or  a 
way  a  system  is  developed.  Constraints  can  be  economic,  political,  technical,  or 
environmental  and  pertain  to  program  resources,  schedule,  environment,  or  to  the 
system  itself. 
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3. 

Are  there  strategies  and  plans  for  evolving  the 

4.1.2 

Modular  Open  Systems  Approach 

4.1.2.C1:  Certain  capabilities,  requirements,  and  program  strategies/objectives  necessitate 

4. 

architecture  while  stabilizing  development  and  providing 

4.2.2 

Management  of  Requirements 

implementing  open  systems  and  developing  open  architectures.  DoDD  5000.1  requires  that  a 

5 

continuity  of  services? 

modular  open  systems  approach  (MOSA)  be  employed  where  feasible.  The  program  should 
identify  open  architecture  enabled  capabilities/objectives  that  reflect  the  following  MOSA 
objectives  (see  the  MOSA  PM  Guide  at  (http://www.acq.osd.mil/osjtf/pmguide.html): 

1.  Facilitate  a  modular  architecture  to  allow  for  affordable  interoperability 

2.  Ensure  a  flexible  and  robust  system  design  to  accommodate  changing  technology  and 
requirements 

3.  Facilitate  integration  with  other  systems  and  use  of  commercial  products  from  multiple 
sources  both  in  the  initial  design  and  in  future  enhancements 

4.  Enable  technology  insertion  as  currently  available  commercial  products  mature  and  new 
commercial  products  become  available  in  the  future 

4.2.2.C10:  The  evolutionary  Acquisition  Strategy  (AS)  utilizes  a  management  system  that 
continually  defines  the  requirements  and  development  activities  to  support  the  evolving  needs; 
adequately  addresses  the  various  concerns  of  users,  developers,  and  managers; 
and  mitigates  the  risks  associated  with  these  issues.  The  basic  system  architecture 
is  designed  to  accommodate  change.  Techniques  such  as  open  systems  design,  functional 
partitioning  and  modular  design  have  been  addressed  by  the  PM  to  achieve  a  flexible 
system  that  can  be  easily  and  affordably  modified. 
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4  Evidence-based  progress  monitoring  and  commitment 

3.3.4 

Management,  Methods,  Metrics,  and 

Sub-Factor  3.3.4.3  -  Technical  Performance  Measures 

reviews 

Techniques 

Pre-Milestone  B  &  C 

4.3.1 

Technical  Review  Planning 

Criteria 

3.3.4.3.C1:  Systems  engineering  uses  technical  performance  measurements  to  balance  cost, 
schedule,  and  performance  throughout  the  life  cycle.  Technical  performance  measurements 
compare  actual  versus  planned  technical  development  and  design.  They  also  report  the  degree  to 
which  system  requirements  are  met  in  terms  of  performance,  cost,  schedule,  and  progress  in 
implementing  risk  handling.  Performance  metrics  are  traceable  to  user-defined  capabilities. 

Factor  4.3.1 -Technical  Review  Planning 

All  Acquisition  Category  (ACAT)  programs  should  include  the  essential  technical  reviews  shown  on 
the  timeline,  as  applicable.  Technical  reviews  provide  a  systematic  process  for  continuously 
assessing  the  technical  baseline,  design  maturity,  technical  risk,  and  programmatic  risk  of 
acquisition  programs.  Technical  reviews  are  consistent  with  existing  and  emerging  commercial 
and  industrial  standards  and  form  the  backbone  of  an  effective  Systems  Engineering  Plan  (SEP). 
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4.1 

Monitoring  of  system  definition  &  development 

2.1 

Program  Schedule  Overview 

"As  the  program  progresses,  the  PM  monitors  the  effectiveness  of  handling  activities  included  in 

progress  vs.  plans 

the  integrated  planning  events  and  schedule  by  comparing  observed  activity  results  with  their 
criteria  and  determining  any  deviations  from  the  planned  schedule." 

95 

4. 

Are  the  levels  and  formality  of  plans,  metrics,  evaluation 

3.3.4 

Management  Methods,  metrics,  and  Techniques 

Not  clear  what  "level  of  project  requirements 

3.3.4.2.C1:  EVM  is  required  on  all  cost  or  incentive  type  acquisition  contracts,  subcontracts,  intra- 

1. 

criteria,  and  associated  mechanisms  (IMP,  IMS,  WBS, 

emergence"  means.  Needs  clarification. 

government  work  agreements,  and  other  agreements  according  to  dollar  thresholds  prescribed  in 

1 

EVMS)  commensurate  with  the  level  of  project 

USD(AT&L)  Policy  Memorandum  dated  March  7,  2005.  The  thresholds  are  as  follows: 

requirements  emergence  and  stability?  (Too  little  is  risky 

In  DAPS,  levels  of  formality  specified  by  size 

•$20  million  or  greater  -  EVM  implementation  compliant  with  ANSI/EIA  -  748  -  A  is  required.  No 

for  pre-specifiable  and  stable  requirements;  too  much  is 

of  project,  not  necessarily  stability  of 

formal  EVM  System  (EVMS)  validation  is  required 

risky  for  emergent  and  unstable  requirements.) 

requirements. 

•$50  million  or  greater  -  EVM  implementation  compliant  with  ANSI/EIA  -  748  -  A  is  required.  An 
EVM  System  must  be  formally  validated  and  accepted  by  the  cognizant  contracting  officer 
•  A  Contract  Performance  Report  (CPR)  and  Integrated  Master  Schedule  (IMS)  are  required 
deliverables  for  all  contracts  that  are  $20  million  or  greater  that  require  EVM 

•  Less  than  $20  million  -  EVM  is  not  required,  except  at  the  discretion  of  the  PM 

(c)  2009  Systems  Engineering  Research  Center 
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4. 

Are  the  project's  staffing  plans  and  buildup  for  progress 

2.1.1 

Program  Schedule/Viability 

Not  sure  what  "buildup  for  progress 

2.1.1.C1: . .  .The  program  manager  (PM)  has  utilized  subject  matter  expertise  of  the  stakeholders 

1. 

monitoring  adequate  with  respect  to  required  levels  of 

2.3.1 

Sufficiency  of  Numbers  and  Qualifications 

monitoring"  means. 

and  the  following  processes  to  develop  the  program  schedule: 

2 

expertise? 

2.1.1. Q26:  What  are  the  changes  in  the  personal  experience  and  subject  matter  expertise  of  the 

IPT  members  involved  in  the  development  of  the  program  schedule? 

2.3.1. Q1:  Is  there  a  staffing  plan  established? 

•  What  is  the  process  to  determine  personnel  resources  and  phasing  required  for  the 
development  of  the  staff,  including  skills,  experience,  and  education  level? 

•  What  are  the  metrics  and  standards  used  to  measure  the  quality  of  the  workforce?  [2.3.1.C1] 

2.3.1. Q18:  What  is  the  experience  level  of  each  of  the  existing  or  planned  key  technical 
personnel? 

•  What  engineering  expertise  is  required  for  the  program? 

•  How  is  the  experience  of  technical  personnel  relevant  to  the  current  activity?  [2.3.1.C3] 
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4. 

Have  most  of  the  planned  project  personnel  billets  been 

2.3.1 

Sufficiency  of  Numbers  and  Qualifications 

2.3.1.Q3:  How  does  the  PMO  describe  the  personnel  issues  affecting  the  program's  ability  to 

1. 

filled  with  staff  possessing  at  least  the  required 

successfully  execute  the  program? 

3 

qualification  level? 

•  What  key  specialties  are  missing? 

•  What  key  billets  are  unfilled/about  to  be  vacated?  [2.3.1.C1] 

2.3.1.Q4:  What  is  the  experience  level  of  each  of  the  existing  or  planned  key  technical  personnel? 
How  is  the  experience  of  technical  personnel  relevant  to  the  current  activity?  [2.3.1.C1] 
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4. 

Is  the  project  adequately  identifying  and  managing  its 

3.3.4 

Management  Methods,  metrics,  and  Techniques 

Sub-Factor  3.3.4.1  -  Risk  Management 

1. 

risks? 

3.3.4.1.C1:  The  Department  of  Defense  (DoD)  recognizes  that  risk  management  is  critical  to 

4 

acquisition  program  success  (see  the  Defense  Acquisition  Guidebook  (DAG). 

3.3.4.1.C2:  There  are  several  notable  changes  of  emphasis  in  the  above  guide  from  previous  RM 
versions.  These  changes  reflect  lessons  learned  from  application  of  risk  management  in  DoD 
programs.  Emphasis  has  been  placed  on: 

•  The  role  and  management  of  future  root  causes, 

•  Distinguishing  between  risk  management  and  issue  management, 

•Tying  risk  likelihood  to  the  root  cause  rather  than  the  consequence, 

•Tracking  the  status  of  risk  mitigation  implementation  versus  risk  tracking,  and 

•Focusing  on  event-driven  technical  reviews  to  help  identify  risk  areas  and  the  effectiveness  of 
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4. 

Have  the  processes  for  conducting  reviews  been 

4.3.1 

Technical  Review  Planning 

States  that  the  reviews  are  held,  "as 

Factor  4.3.1  -  Technical  Review  Planning 

1. 

evaluated  for  feasibility,  reasonableness,  completeness. 

1.3.1 

Reasonableness,  Stability,  and  Testability 

applicable" 

All  Acquisition  Category  (ACAT)  programs  should  include  the  essential  technical  reviews  shown  on 

5 

and  assurance  of  independence? 

4.1.1 

System  Assurance 

Independent  assessments  are  performed 

the  timeline,  as  applicable.  Technical  reviews  provide  a  systematic  process  for  continuously 
assessing  the  technical  baseline,  design  maturity,  technical  risk,  and  programmatic  risk  of 

throughout,  but  the  contractor's  processes 

acquisition  programs. 

for  conducting  reviews  is  not  really  addressed 

1.3.1.C14:  Computer/software  configuration  items  have  completed  test  verification,  and  the 

except  by  the  SEP. 

system  software  capability  is  determined  to  be  mature.  All  known  deficiencies  have  been 
documented  and  evaluated,  and  fixes  have  been  identified  and  rescheduled  for  verification.  An 
Independent  Verification  and  Validation  (IV&V)  assessment  of  the  contractor/materiel  developer 
has  been  performed. 

4.3.1. C3:  The  ITR  ensures  that  a  program's  technical  baseline  is  sufficiently  rigorous  to  support  a 
valid  cost  estimate  (with  acceptable  cost  risk),  and  enable  an  independent  assessment  of  that 
estimate  by  cost,  technical,  and  program  management  subject  matter  experts. 

4.3.1. C15:  The  SRR  is  typically  conducted  by  a  technical  review  board  consisting  of  a 
government  chairperson  selected  outside  (independent  of)  the  government  program  office. 

4.3.1. Q32:  For  ACAT  ID  or  1AM  programs,  the  service  acquisition  official  provides  a 
recommendation  to  the  Director,  Defense  Research  and  Engineering  (DDR&E)  of  the  Office 
of  the  Secretary  of  Defense  for  Deputy  Under  Secretary  of  Defense  for  Science  and 

Technology  (DUSD(S&T))  final  approval.  If  deemed  necessary,  the  DDR&E  can  conduct  an 
Independent  Technical  Assessment  (ITA)  in  addition  to,  and  totally  separate  from,  the  TRA: 

4.3.1. C56:  Prior  to  the  OTRR  the  OUSD(AT&L)  Systems  and  Software  Engineering/ 

Assessments  and  Support  (SSE/AS)  staff  will  conduct  an  assessment  of  operational  test 
readiness  (AOTR)  to  independently  assess  the  successful  completion  of  developmental  test 
and  evaluation  (DT&E)  and  report  the  AOTR  findings  to  the  PM  and  Deputy,  OUSD(A&T). 

4.4.2. C7:  Government  and  contractor  use  common  M&S  tools  to  support  both  development 
and  test  and  evaluation.  Simulations  used  to  evaluate  program  performance  as  part  of  the  test 
and  evaluation  process  are  verified  independently  from  contractor  simulations  and  undergo  the 
same  level  of  verification,  validation,  and  accreditation  (VV&A). 

4.5.2. Q3:  How  will  the  various  program  estimates  be  vetted?  How  similar/different  are  the 
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4. 

Has  compliance  with  legal,  policy,  regulatory,  standards, 

3.2.1 

Statutory  and  Regulatory  Compliance  and 

This  question  seems  too  vague.  It  assumes  a 

3.2.1.Q10:  Does  the  PMO  have  a  clear  and  concise  understanding  of  all  DoD  and  Service-level 

1. 

and  security  requirements  been  clearly  demonstrated? 

Guidance 

lot  of  knowledge  and  can  pertain  to  reporting 

policies  and  statutes  that  the  program  must  comply  with?  [3.2.1.C4] 

6 

3.2.3 

Certifications 

mechanisms,  processes,  government  and 

3.2.1.Q11:  Have  the  following  statutory  information  requirements  been  met?  Who  is  the  approval 

3.4 

Contract  Management 

contract  conduct,  etc.  It  also  has  to  be  taken 

authority  and  what  is  the  approval  date?  Note:  See  DAG  for  applicable  statutes  for  each 

in  perspective  of  the  life  cycle. 

information  requirement. 

•Consideration  of  technology  issues 

101 
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2.1.1. Q9:  What  is  the  process  established  to  monitor  program  performance  through  the  schedule? 
•Are  the  following  identified? 

-Key  events 
-  Milestones 
-Reviews 

-All  integrated  technical  tasks 

-Accomplishment  criteria  and  schedule  metrics  [2.1.1.C2] 

2.1.1. Q17:  What  is  the  highest  risk  path,  both  for  the  overall  program  schedule  and  for  the  SDD 
schedule? 

•How  has  the  PM  applied  resources  against  the  activities  on  this  risk  path?  [2.1.1.C3  and  2.1.1.C4] 

3.2.2. Q32:  In  preparation  for  the  SRR,  were  the  following  actions  completed? 

•  Successful  completion  of  all  post-award  activities 

•  Published  agenda  (several  weeks  prior  to  the  conference  -  to  permit  sufficient  time  for 
government  preparation 

•  Draft  system  specification  and  any  initial  draft  performance  item  specifications 

•  Functional  analysis  (top  level  block  diagrams) 

•  Feasibility  analysis  (results  of  technology  assessments  and  trade  studies  to  justify  system  design 
approach) 

3.3.4.1. C1:  The  Department  of  Defense  (DoD)  recognizes  that  risk  management  is  critical  to 
acquisition  program  success  (see  the  Defense  Acquisition  Guidebook  (DAG).  The  purpose  of 
addressing  risk  on  programs  is  to  help  ensure  program  cost,  schedule,  and  performance  objectives 
are  achieved  at  every  stage  in  the  life  cycle  and  to  communicate  to  all  stakeholders  the  process 
for  uncovering,  determining  the  scope  of,  and  managing 

program  uncertainties.  Since  risk  can  be  associated  with  all  aspects  of  a  program,  it  is 
important  to  recognize  that  risk  identification  is  part  of  the  job  of  everyone  and  not  just 
the  program  manager  or  systems  engineer.  That  includes  the  test  manager,  financial 
manager,  contracting  officer,  logistician,  and  every  other  team  member. 

1.2.1. Q1:  How  were  mission  tasks  (MTs),  measures  of  effectiveness  (MOEs),  and  measures  of 
performance  (MOPs)  derived  from  relevant  guidance  on  requirements  or  capabilities  (e.g.. 

Mission  Needs  Statement  (MNS),  Operational  Requirements  Document  (ORD)  (if  pertinent),  or  the 
problem  statement  found  in  the  ICD?  [1.2.1.C1] 

•Are  they  quantifiable?  [1.2.1.C1] 

1.2.1. Q7:  What  are  the  models  and  simulations  used  in  the  study? 

•Are  they  acceptable  and  accredited?  [1.2.1.C1] 

1.2.2. C1:  The  Analysis  of  Alternatives  (AoA)  study  plan  describes  a  clear  link  between  the  AoA, 
capability  needs,  system  requirements,  and  the  measures  of  effectiveness  (MOEs)  used  to 
evaluate  the  system(s). 

1.2.2. Q13:  How  does  the  program  use  realistic  and  current  architectures  and  the  CONOPS  to 
derive  alternative  solutions  and  to  ensure  a  clear  understanding  of  potential  C4I  interfaces  and 
interoperability  needed  during  military  operations? 

•Initial  Capabilities  Document  (ICD) . . .  supports  the  Analysis  of  Alternatives  (AoA),  Technology 
Development  Strategy  (TDS),  Milestone  A  decisions,  and  subsequent  Technical  Development  (TD) 
phase  activities. 

2.2.1. Q14:  What  were  the  results  of  the  Alternative  System  Review  (ASR)? 

•  Did  the  IPT  determine  that  the  operational  capabilities,  preferred  solution(s),  available 
technologies,  and  program  resources  (funding,  schedule,  staffing,  and  processes)  form 

a  satisfactory  basis  for  proceeding  into  the  TD  phase? 

•Is  the  program  schedule  executable  (technical  and/or  cost  risks)?  [2.2.1.C2] 

3.1.1. Q23:  How  did  the  Acquisition  Strategy  address  risk  management? 

•What  statistical  or  other  qualitative  procedures  were  followed  to  "measure"  program 
risk? 

•What  is  the  risk  management  structure  for  selecting  acquisition  alternatives?  [3.1.1.C3] 

3.4.3. Q23:  How  did  the  government  and  contractor,  through  the  VE  process,  analyze  the 
essential  requirements,  military  and  technical  characteristics,  and  the  design  tasks  to 
develop  possible  alternatives  offering  improved  value? 
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4. 

Has  the  project  identified  a  full  set  of  representative 

1.2.1 

Validity  and  Currency 

1.2.1.Q1:  How  were  mission  tasks  (MTs),  measures  of  effectiveness  (MOEs),  and  measures  of 

2. 

operational  scenarios  across  which  to  evaluate 

4.1.8 

Human  Systems  Integration 

performance  (MOPs)  derived  from  relevant  guidance  on  requirements  or  capabilities  (e.g.. 

3 

feasibility? 

4.4.2 

Modeling  and  Simulatioin  Tools 

Mission  Needs  Statement  (MNS),  Operational  Requirements  Document  (ORD)  (if  pertinent),  or  the 
problem  statement  found  in  the  ICD?  [1.2.1.C1] 

•Are  they  quantifiable?  [1.2.1.C1] 

1.2.1. Q2:  Are  the  MOEs  stated  in  terms  of  military  utility  and  based  on  value  provided  to  the 
warfighter? 

•Are  these  MOEs  used  to  identify  models,  simulations,  and  other  analysis  tools  required  to 
execute  the  study?  [1.2.1.C1] 

1.2.1. Q3:  What  are  the  relevant  issues  and  constraints  as  addressed  in  the  study  plan?  [1.2.1.C1] 

1.2.1. Q4:  Is  the  range  of  alternatives  comprehensive?  [1.2.1.C1] 

1.2.1. Q5:  Are  the  threats  and  scenarios  realistic  and  current?  [1.2.1. Cl] 

1.2.1. Q15:  Were  the  threats  and  scenarios  used  in  the  study  appropriate  and  approved  by  Defense 
Intelligence  Agency  (DIA)? 

4.1.8:  •  Are  scenario-based  factors  such  as  environmental  conditions,  conflict  durations,  etc. 
included?  [4.1.8.C1,  C2] 

4.4.2. C5:  The  program  has  a  plan  to  acquire  domain  knowledge  for  each  M&S  objective- 
scenario  set.  This  domain  knowledge  includes  the  entities,  attributes  and  interactions 
that  have  significant  bearing  on  the  objective  at  the  level  of  resolution  and  fidelity 
required  for  the  effort. 
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4. 

Has  the  project  prepared  milestone  plans  and  earned 

3.3.1 

Program  Plan/Schedule 

Unclear  if  this  means  establishing  thresholds 

3.3.1.C7:  The  program  has  appropriate  development  activities  planned  and  scheduled,  e.g. 

2. 

value  targets  for  measuring  progress  in  developing 

for  the  schedule  and  cost  metrics.  Target 

Integrated  Master  Plan/Integrated  Master  Schedule  (IMP/IMS),  and  implements  these  activities  to 

4 

feasibility  evidence? 

values  are  not  discussed  in  DAPS 

execute  the  program.  These  planned  and  scheduled  activities  include  completion  criteria.  Program 
funding  and  schedules  are  sufficient  to  accommodate  technical  complexity  and  identified  program 
risks.  Sufficient  resources  are  allocated  and  available  to  the  program  to  successfully  develop  the 
system  within  the  program  baseline. 

3.3.1.C8:  The  program  is  following  the  program  management  plans  in  executing  the  program.  The 
program  has  accomplished/is  accomplishing  the  planned  activities  with  minimal  schedule  impact 
and  is  proceeding  to  execute  within  the  program  baselines.  Schedule  performance  is  reported 
through  an  Earned  Value  Management  System  (EVMS). 
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4. 

Is  the  project  successfully  monitoring  progress  and 

3.4.1 

Contracting/Prime  Contractor  Management 

3.4.1.C5b:  Design  and  Production  Assurance  -  monitor  the  performance  of  the  contractor  against 

2. 

applying  corrective  action  where  necessary? 

Test  and  Evaluation  Plan 

contract  requirements  to  enable  timely  corrective  action. 

5 

4.6.1 

3.4.1.Q74:  In  terms  of  Integrated  Baseline  Reviews  (IBRs): 

•  How  did  the  contractor  address  the  government's  intent  to  conduct  IBRs  after  contract  award? 

•  Who  developed  the  guidelines,  criteria,  and  processes  for  the  IBR? 

•  Who  lead  the  technical  assessments  during  IBRs? 

•  Upon  completion,  how  are  the  results  of  the  IBR  documented  and  provided  to  appropriate  team 

members? 

•  What  action  plan  is  prepared  to  correct  any  problem  areas  discovered  during  the  review? 

•What  is  the  process  to  track  corrective  actions  and  interfaces  with  the  contractor  during  program 

reviews  until  the  corrective  actions  are  completed?  [3.4.1.C5b] 

4.6.1.C11:  A  Failure  Reporting,  Analysis  and  Corrective  Action  System  (FRACAS)  has  been  initiated. 
The  systems  engineering  (SE)  process  provides  tracking  between  test  activities  and  technical 
requirements. 

•  Discuss  the  planned  FRACAS  program.  What  is  the  planned  time  for  root  cause  analysis  and 
corrective  action  for  major  and  minor  hardware/software  deficiencies? 

[4.6.1.C19,  4.6.1.C20,  and  4.6.1.C22] 
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4.3 

Monitoring,  assessment,  and  replanning  for  changes  in 
needs,  opportunities,  and  resources 

(c)  2009  Systems  Engineering  Research  Center 
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4. 

3. 

1 

Does  the  project  have  an  effective  strategy  for 
performing  triage  (accept,  defer,  reject)  on  proposed 
changes,  which  does  not  destabilize  ongoing 
development? 

1.3.1 

2.1.1 

3.1.1 

3.2.2 

3.3.1 

3.3.6 

3.4.1 

Reasonableness,  Stability,  and  Testability 

Program  Schedule/Viability 

Acquistion  Strategy/Credibility 

Entrance  and  Exit/Success  Criteria 

Program  Plan/Schedule 

management  of  Dependencies  and  External 

Interfaces 

Prime  contractor  Management 

DAPS  discusses  allowing  for  and  managing 
change  but  does  not  present  methods  for 
doing  so  like  triage. 

1.3.1. Q8:  Were  any  changes  made  to  the  ICD  between  JROC  approval  and  the  Milestone  A 
decision  review? 

•How  were  these  changes  vetted  through  the  requirements  generation  and  acquisition 
management  processes? 

1.3.1. Q17:  What  controls  are  in  place  to  prevent  "requirements  creep"  and  to  force  new 
requirements  to  be  defined  through  an  engineering  change  proposal  process?  [1.3.1.C5] 

2.1.1. C1:  The  program's  overall  schedule  is  viable  (i.e.,  workable  and  has  real  meaning  and 
pertinence).. . . 

•  Implementation  of  procedure(s)  to  control  changes  to  the  program  schedule.. . . 

•Revision  of  the  procedure(s)  to  control  changes  to  the  program  schedule,  if  required. 

2.2.1. Q4:  What  is  the  PM's  process  to  prevent  unexpected  or  unplanned  cost  growth  by 
adequately  identifying  and  managing  risks  in  the  program? 

•  What  is  the  process  to  allocate  funding  (level  and  timeliness)  to  cover: 

-  Systems  Engineering  (SE)  technical  reviews 

-  Risk  mitigation 

-  Engineering  changes 

3.1.1. Q4:  Are  any  of  the  potential  causes  of  instability  to  the  Acquisition  Strategy  present?  If  so, 
how  is  the  PM  working  to  mitigate  the  impact  to  the  program's  Acquisition  Strategy? 

3.1.1. Q5:  How  is  the  PM  emphasizing  the  following  "aids"  to  a  stable  Acquisition  Strategy? 

•  Direction. 

•  Advocacy. 

•  Commitment. 

•Use  of  Integrated  Product  Teams. 

3.3.1. Q9  What  are  the  in  place  processes  to  manage  the  Technology  Development  (TD)  phase  effoi 
3.3.6.Q15:  How  are  changes  in  SoS  constituent  systems  negotiated  with  their  PMs?  [3.3.6.C14] 

3.4.1. C5:  The  program's  third  phase  -  execution  and  sustainment  -  is  being  successfully  completed 

3.4.1. Q71:  How  are  Engineering  Change  Proposals  (ECPs)  and  alterations  affecting  cost  and  schedu 

•  Are  the  changes  within  the  scope  of  the  contract? 

•Was  pricing  information  to  support  the  ECP  requested  from  the  contractor? 
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4. 

3. 

2 

Does  the  project  have  an  adequate  capability  for 
performing  change  impact  analysis  and  involving 
appropriate  stakeholders  in  addressing  and  prioritizing 
changes? 

1.3.1 

2.1.1 

3.1.1 

3.2.2 

3.3.1 

3.3.6 

3.4.1 

Reasonableness,  Stability,  and  Testability 

Program  Schedule/Viability 

Acquistion  Strategy/Credibility 

Entrance  and  Exit/Success  Criteria 

Program  Plan/Schedule 

management  of  Dependencies  and  External 

Interfaces 

Prime  contractor  Management 

3.2.2.Q61:  In  preparation  for  the  PRR,  were  the  following  activities/actions  completed  or 
documents/information  available? 

•Provisions  have  been  made  for  determining  producibility  and  cost  impacts  of  engineering 
changes  introduced  during  production 

3.4.1. C5:  The  program's  third  phase  -  execution  and  sustainment  -  is  being  successfully  completed 
through  insight  into  program  progress,  and  the  effective  management  of  the  impact  of  changes, 
whether  these  changes  are  due  to  contract  execution  or  to  external  influences.  As  the  program 
progresses,  the  PM  makes  viable  and  timely  decisions  and  provides  direction  to  accommodate 
changing  circumstances.  Focus  is  maintained  on  the  risk  areas  most  likely  to  impact  the  program. 
The  PM  uses  those  indicators  developed  in  the  previous  stages,  i.e.,  EVMS,  IMS  and  appropriate 
metrics,  for  primary  program  insight. 

4.2.2. Q6:  How  is  the  requirements  management  process  during  TD  supported  by  the  resource 
management  tools? 

•When  changes  are  made,  how  are  the  impacted  requirements  identified  and  accounted  for  in 
the  updated  system?  [4.2.2. C8] 

4.2.3. Q3:  For  a  system  of  systems  (SoS)  and  family  of  systems  (FoS),  what  is  the 
process  used  to  assess  the  impact  of  incorporating  a  new  capability  within  the 
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4. 

3. 

3 

Is  the  project  adequately  verifying  and  validating 
proposed  changes  for  feasibility  and  cost-effectiveness? 

1.3.1 

2.1.1 

3.1.1 

3.2.2 

3.3.1 

3.3.6 

3.4.1 

Reasonableness,  Stability,  and  Testability 

Program  Schedule/Viability 

Acquistion  Strategy/Credibility 

Entrance  and  Exit/Success  Criteria 

Program  Plan/Schedule 

management  of  Dependencies  and  External 

Interfaces 

See  above. 
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4.5 

Use  of  milestone  reviews  to  ensure  stakeholder 

commitment  to  proceed 

(c)  2009  Systems  Engineering  Research  Center 
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4. 

5. 

1 

Are  milestone  review  dates  based  on  the  availability  of 
feasibility  evidence,  instead  of  the  availability  of  artifacts 
or  the  occurrence  of  planned  review  dates? 

3.2 

3.2.2 

3.3.1 

Knowledge-based  Decisions  and  Milestones 
Entrance  and  Exit/Success  Criteria 

Program  Plan/Schedule 

The  DAPS  specifies  reviews  based  on 
entrance  and  exit  criteria.  These  entrance 

and  exit  criteria  are  activities  that  need  to  be 
accomplished  (e.g.,  requirements  are 
traceable...)  not  schedule  or  artifacts.  Same 
with  the  master  plan  and  schedule. 

Knowledge-based  acquisition  is  a  management  approach  that  requires  adequate  knowledge  at 
critical  junctures  (i.e.,  knowledge  points)  throughout  the  acquisition  process  to  make  informed 
decisions. 

The  following  knowledge  points  coincide  with  decisions  along  the  acquisition  framework: 

•  Entrance  Criteria:  Each  phase  has  defined  entrance  criteria  that  are  based  on  the  definition  and 
validation  of  needed  capabilities,  technology  maturity,  system  design  maturation,  and  funding. 
Major  decision  points  (e.g.  MS  B,  C)  mark  the  entrance  into  succeeding  phases,  with  specific 
decision  points  tailored  on  a  program-by-program  basis  and  supported  by  technical  and 
programmatic  reviews. 

3.2.2. C7:  Entrance  Criteria  into  technical  and  programmatic  reviews  (e.g.,  IBR,  SRR,  System 
Functional  Review  (SFR),  Software  Specification  Review  (SSR),  Preliminary  Design  Review  (PDR), 
Design  Readiness  Review  (DRR),  Critical  Design  Review  (CDR),  Test  Readiness  Review  (TRR), 

System  Verification  Review  (SVR),  Production  Readiness  Review  (PRR),  and  TRA),  in  support  of 
specific  decision  points  conducted  during  the  SDD  phase,  have  been 

successfully  met. 

3.2.2. C8:  Exit/Success  Criteria  from  TD  phase:  Successful  development,  maturation,  and 
evaluation  of  the  technologies  needed  for  the  capability  under  consideration.  The 
maturation  of  the  required  technologies  is  consistent  with  the  prescribed  Technology 

Readiness  Levels  (TRLs). 

3.2.2. C9:  Exit/Success  Criteria  for  technical  and  programmatic  reviews  conducted  during 

TD  phase  (i.e.,  SRR,  IBR  and  TRA),  in  support  of  decision  points  were  successfully 
conducted  with  valid  documentation,  data,  and  analyses. 

3.3.1.C1:  The  Integrated  Master  Plan  (IMP)  is  an  event-driven  plan  that  documents  the 
significant  accomplishments  necessary  to  complete  the  work  and  ties  each 
accomplishment  to  a  key  program  event  that  forms  the  foundation  of  the  Integrated 

Master  schedule  (IMS).  Note:  The  IMP  Events  are  not  tied  to  calendar  dates;  each  event 
is  completed  when  its  supporting  Accomplishments  are  completed  and  as  evidenced  by 
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4. 

5. 

2 

Are  artifacts  and  evidence  of  feasibility  evaluated,  and 
risky  shortfalls  identified,  by  key  stakeholders  and 
independent  experts,  prior  to  review  events? 

2.2.1 

3.1.1 

4.5.1 

Program  Funding  and  Allocation 

Acquisition  Strategy/Credibility 

Software  Development  Plan 

The  DAPS  discusses  this  in  detail  for  every 
review.  However,  it  does  require  that 
artifacts  be  submitted  to  the  government  in 
preparation  for  the  review.  Entrance  and 

Exit  Criteria  for  reviews  are  not  discussed  in 

detail 

•Initial  Capabilities  Document  (ICD).  The  ICD  replaced  the  Mission  Needs  Statement.  The  ICD 
documents  the  need  for  a  materiel  approach  to  a  specific  capability  gap  derived  from  an  initial 
analysis  of  materiel  approaches  executed  by  the  operational  user  and,  as  required,  an 
independent  analysis  of  materiel  alternatives. 

2.2.1. Q11:  What  were  the  results  of  the  Initial  Technical  Review  (ITR)? 

•Is  the  program's  technical  baseline  sufficiently  rigorous  to  support  a  valid  cost  estimate  (with 
acceptable  cost  risk)? 

•How  does  it  enable  an  independent  assessment  of  the  estimate  by  cost,  technical,  and  program 
management  subject  matter  experts? 

3.1.1. Q34:  How  does  the  Acquisition  Strategy  address  key  aspects,  including  risks,  of  the  proposed 
software  development  approach? 

•Does  it  state  how  the  chosen  software  development  approach  supports  the  system-level 
Acquisition  Strategy? 

•What  is  the  plan  for  using  independent  expert  reviews  for  a  software-intensive  program? 
[3.1.1.C3] 

Design  Considerations 

4.5.1. C18:  The  software  development  has  followed  a  disciplined  process  documented  in  the 
program  SDP  and  related  plans.  This  process  includes  reviews,  design, 
implementation  and  integration  and  test.  Reviews  have  proceeded  based  on  documented 
entrance  and  exit  criteria  and  results  are  captured  in  minutes  and  updates  to  plans,  artifacts, 
design,  and  code.  Tools  and  facilities  exist  and  are  used  to  execute  the  software 
development  and  verification  (testing).  The  current  status  of  software  completion  verification 
testing  is  consistent  with  the  verification  test  schedule. 

3.2.2. Q30:  In  preparation  for  the  IBR,  were  the  following  documents  provided  by  the  contractor 
to  the  government  for  review?  (SOW,  WBS,  CWBS,  CAPs  . . .) 
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4. 

5. 

3 

Are  developer  responses  to  identified  risks  prepared 
prior  to  review  events? 

3.2.2 

Entrance  and  Exit/Success  Criteria 

Identifying  risks  is  mentioned  as  entrance 
criteria,  but  it  does  not  specify  that  developer 
responses  must  be  prepared  prior  to  review 
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4. 

5. 

4 

Do  reviews  achieve  risk-based  concurrence  of  key 
stakeholders  on  whether  to  proceed  into  the  next 
phase?  (Proceed;  skip  a  phase;  revisit  the  current  phase; 

3.2.2 

4.3 

Entrance  and  Exit/Success  Criteria 

Technical  Baselines 

DAPS  process  ensures  stakeholder 
review/concurrence  at  all  major  milestones 

Figure  4.3  Systems  Engineering  Technical  Review  Timing 

(c)  2009  Systems  Engineering  Research  Center 
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APPENDIX  F:  SERC  SE  EM:  PROPOSED  NEW  FRAMEWORK 

2/16/2009 

At  our  January  29-30  Workshop,  we  encountered  three  candidate  frameworks  for  organizing  our 
systems  engineering  (SysE)  effectiveness  measures  (EMs): 

1.  The  5x5  matrix  based  on  the  DoD  SysE  Plan  Preparation  Guide  and  included  in  our  EM  task 
proposal,  for  which  we  had  prepared  in  instrument  for  evaluating  the  functional  coverage  of  the 
8  Candidate  EM  approaches  we  are  in  the  process  of  evaluating. 

2.  Another  5x5  matrix  developed  by  the  US -UK- Australia  Software-Intensive  Systems  Acquisition 
Improvement  Group  (SISAIG),  to  serve  as  a  review  framework  for  identifying  early  warning 
indicators  for  troubled  projects.  It  was  expanded  by  USC  into  a  Macro  Risk  Tool  for  NASA 
projects,  in  which  each  of  the  25  elements  have  a  set  of  subsidiary  questions  about  sources  of 
project  risk.  The  tool  is  tailorable  to  different  projects  by  assigning  different  weights  to  the 
questions.  It  was  proposed  at  the  January  Workshop  as  a  tool  framework  for  projects  to  use  in 
applying  the  end-result  EMs  from  the  SERC  task  to  DoD  project  reviews. 

3.  The  45-row  Candidate  EM  Coverage  Matrix  providing  an  initial  USC  assessment  of  which 
individual  EMs  were  covered  by  which  Candidate  EM  approaches.  It  was  found  at  the 
Workshop  to  be  a  good  way  to  compare  the  candidate  EM  approaches,  subject  to  having  the  EM 
approach  originators  update  the  USC  assessments,  and  subject  to  finding  a  better  organization 
for  the  45  items. 

After  the  Workshop,  we  perfonned  crosswalks  among  the  three  frameworks,  and  synthesized  the 
proposed  new  framework  shown  on  the  next  page.  It  is  a  bit  simpler  than  the  5x5s,  having  4  major 
categories  and  4-5  elements  per  category.  It  also  identifies  the  counterpart  EM  items  in  the  three 
frameworks  above,  showing  both  their  overlaps  and  their  candidate  subsidiary  questions  to  ask  about 
sources  of  project  risk.  The  category  elements  are  not  mutually  exclusive  (topics  such  as  COTS  and 
reuse  arise  in  several  contexts,  for  example),  but  they  are  reasonably  orthogonal,  and  they  are  exhaustive 
in  that  they  cover  all  of  the  EM  items  in  the  three  frameworks  above. 

We  propose  to  use  the  new  framework  as  the  organizing  principle  for  a  revised  SysE  EM  Macro  Risk 
Tool.  However,  we  would  propose  to  proceed  to  use  the  Coverage  Matrix  in  doing  each  team  member’s 
assessments  of  the  strength  of  each  Candidate  EM  approach  in  addressing  each  Coverage  Matrix 
element  on  a  Green- Yellow-Orange-Red  basis  as  was  done  in  Joe  Elm’s  and  Paul  Componation’s  self- 
assessments.  Once  we  discuss  and  reconcile  or  note  differences  in  evaluators’  ratings,  we  can  then 
populate  the  Coverage  Matrix  items  into  the  new  EM  framework,  and  revise  the  Macro  Risk  Tool  to 
serve  as  a  review-oriented  EM  evaluation  tool. 
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Systems  Engineering  Effectiveness  Measurement 
Proposed  New  Framework 

SEPP-Guide- 
Based  Eval. 
Framework 

SISAIG/ 
Macro  Risk 
Framework 

Coverage 
Matrix  Items 

1 .  Concurrent  Definition  of  System  Requirements  &  Solutions 

1.1  Understanding  of  stakeholder  needs:  Capabilities, 
Operational  Concept,  Key  Performance  Parameters, 
Enterprise  fit  (legacy) 

1.1,  1.4, 

3.1 

1.1,  1.4 

5,  7,  22, 

36,  37 

1.2  Concurrent  exploration  of  solution  opportunities;  AoA’s  for 
cost-effectiveness  &  risk  (Measures  of  Effectiveness) 

4.1,  4.2 

1.2 

1,  14,26, 
27,  28 

1.3  System  scoping  &  requirements  definition  (External 
interfaces;  Memoranda  of  Agreement) 

1.2,  1.4 

3.2 

4,  6,  13, 

50 

1 .4  Prioritization  of  requirements  &  allocation  to  increments 

1.3 

1.5 

2,  11,  31 

2.  System  Life  Cycle  Organization,  Planning,  Staffing 

2.1  Establishment  of  stakeholder  Life  Cycle  Responsibility, 
Authority,  and  Accountabilities  (RAAs)  (for  System 
Definition,  System  Development,  System  Operation) 

2.1 

2.1,  2.3, 
2.5 

2,  17,  20, 
46 

2.2  Establishment  of  Integrated  Product  Team  (IPT)  RAAs, 
Cross-IPT  coordination  needs 

2.2 

2.2,  2.4 

32 

2.3  Establishment  of  necessary  resources  for  meeting 
objectives 

3.5,  4.2, 
4.6 

2.4 

9,  40 

2.4  Establishment  and  usage  support  of  appropriate  source 
selection,  contracting,  &  incentive  structures 

2.1 

2.1,  2.5 

21,33,42 

2.5  Assurance  of  necessary  personnel  competencies 

3.2,  3.3, 
3.4 

2.4,  2.6 

16,  19,  20, 
23 

3.  Technology  Maturing,  Architecting 

3.1  COTS/NDI/Services  evaluation,  selection,  validation  for 
capability,  maturity  &  compatibility 

4.5 

3.5 

28 

3.2  Life  Cycle  architecture  definition  &  validation 

4.1,  4.2 

1.2,  3.2, 
3.4 

1,  12,  13, 
14,  30,  39, 
44 

99 


3.3  Use  of  prototypes,  exercises,  models,  and  simulations  to 
determine  technology  maturity,  architecture  feasibility 

4.3,  4.5 

3.3,  3.5 

3,  10,  15, 
26,  27,  28 

3.4  Validated  System  Engineering,  Development, 

Manufacturing,  Operations  &  Maintenance  budgets  & 
schedules 

4.4 

3.3,  5.1 

8,  9,  18 

4.  Evidence-Based  Progress  Monitoring  &  Commitment  Reviews 

4.1  Monitoring  of  system  definition  &  technical  development 
progress  vs.  plans 

5.1,  5.2 

4.4,  4.5, 
5.2,  5.5 

23,  24,  25, 
29,38,41, 
43,  44,  45 

4.2  Monitoring  of  feasibility  evidence  development  progress 
vs.  plans 

5.3 

3.3,  5.4 

8,  26,  27, 
29,  30,  49 

4.3  Monitoring,  assessment,  and  replanning  for  changes  in 
needs,  opportunities,  and  resources 

2.2,  5.4, 
5.6 

1.5,  5.3 

30,  34,  35, 
40,  41 

4.4  Identification  and  mitigation  planning  for  feasibility 
evidence  shortfalls  and  other  risks 

4.3,  5.2, 

5.3 

1.2,  5.4 

8,  15,47, 
48 

4.5  Use  of  milestone  reviews  to  ensure  stakeholder 
commitment  to  proceed 

4.6 

4.1,  4.2, 
4.3,  4.4, 
4.5 

9,  51 
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APPENDIX  G:  EVOLUTION  OF  THE  EM  PROJECT  PLANS  AND 

SCHEDULES 


1.  Sponsor  Guidance 

The  initial  guidance  provided  in  the  Request  for  Proposal  and  contract  Statement  of  Work  was  to 
develop  SE  EMs  suitable  for  use  by  contractor  management,  DoD  program  managers,  and  DoD 
oversight  officials  in  evaluating  the  effectiveness  of  a  project’s  systems  engineering  activities  across  the 
range  of  weapons  platforms,  systems  of  systems,  and  net-centric  services.  The  final  guidance  provided 
by  the  ultimate  task  sponsors  can  be  summarized  as  to:  Focus  Initial  Task  Pilot  Evaluations  and 
Recommendations  on  Major  Defense  Acquisition  Programs  (MDAPs);  Develop  and  Iterate  a  Coverage 
Matrix;  Develop  a  Framework,  Operational  Concept,  and  Tools;  and  to  relate  these  to  the  Defense 
Acquisition  Program  Support  (DAPS)  Methodology 

Our  Office  of  the  Under  Secretary  of  Defense  (Acquisition,  Technology,  and  Logistics) 
(OUSD(AT&L))  sponsors  initially  saw  the  greatest  near-term  need  for  EMs  in  the  Weapons  Platfonn 
domain.  They  asked  us  to  structure  the  evaluations  to  enable  them  to  be  used  in  potential  follow-on 
tasks  for  the  System  of  Systems  and  Net-Centric  Services  domains,  but  to  focus  our  pilot  evaluations 
and  recommendations  on  Weapons  Platfonns.  This  changed  each  team  member’s  Statement  of  Work 
accordingly,  and  initially  enabled  us  to  develop  more  thorough  results  for  the  Weapons  Platform 
domain. 

In  a  subsequent  telephone  conference  on  January  9,  our  sponsors  indicated  that  they  would  like  to  see 
more  up-front  work  done  on  the  survey  and  interviews,  and  on  a  coverage  matrix  indicating  which  EMs 
cover  which  individual  candidate  measures  of  effectiveness.  They  preferred  to  have  us  focus  a  planned 
January  29-30  workshop  on  a  SERC-internal  assessment  of  progress  to  date  on  these,  to  defer  the 
SERC-extemal  workshop  involving  potential  EM  users  and  pilot  candidates  until  late  March-early  April 
(March  30-April  3  week  proposed)  and  to  do  one  round  of  pilot  evaluations  rather  than  two.  The 
University  of  Southern  California,  USC,  developed  a  framework  and  perfonned  the  coverage  matrix 
subtask. 

At  the  January  29-30  workshop,  the  coverage  matrix  was  found  to  provide  a  good  basis  both  for 
comparison  of  the  candidate  EMs  and  as  an  improved  framework  for  EM  evaluation,  subject  to  having 
the  EM  originators  iterate  the  USC  assessments  of  their  coverage,  adding  a  strength-of-coverage  rating 
level,  and  reorganizing  the  coverage  list  into  an  evaluation  framework.  USC  therefore  revised  the 
framework  and  the  evaluation  instrument.  Based  on  further  initial  sponsor  guidance,  our  plans  were 
revised  to  add  a  Personnel  Competency  EM,  to  reassign  Massachusetts  Institute  of  Technology  (MIT)’s 
tasks,  due  to  MIT’s  inability  to  participate  in  the  SERC,  and  to  show  the  new  schedule  of  activities  and 
results. 

At  the  May  6  workshop  and  a  follow-on  May  7  meeting,  the  sponsors  indicated  that  it  would  be  valuable 
to  relate  the  EM  framework  to  the  DAPS  Methodology,  and  to  perfonn  the  EM  evaluation  with  respect 
to  the  DoD  Systemic  Analysis  Database  (SADB)  by  furnishing  questions  to  the  SADB  proprietors  rather 
than  receiving  a  Government  Furnished  Information  (GFI)  version  of  the  SADB.  They  also  indicated 
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that  since  the  framework  and  tools  appeared  to  apply  to  Major  Defense  Acquisition  Programs  (MDAPs), 
we  should  extend  the  scope  from  weapons  platforms  to  MDAPs.  We  revised  our  plans  accordingly, 
along  with  accommodating  the  no-cost  extension  of  the  task  to  30  September  2009. 


2.  Initial  Key  Activities 

Our  initial  key  activities  included  creating  a  revised  list  of  candidate  EMs  to  evaluate;  adding  a 
Personnel  Competency  EM;  reassigning  the  MIT  evaluations. 

Given  the  focus  on  Weapons  Platforms,  discussions  with  our  initial  sponsors  and  with  developers  of 
candidate  EMs,  and  additional  candidate  EMs  that  emerged  since  the  proposal,  we  added  three 
promising  candidate  EMs.  To  balance  the  workload  and  to  focus  our  work  on  the  strongest  and  most¬ 
relevant  EMs,  we  also  dropped  five  less-strong  or  less-relevant  candidate  EMs  that  were  in  our  proposal. 
Below  are  the  EM  candidates  added  and  dropped,  with  short  comments  on  each. 

At  the  January  29-30  workshop,  our  sponsor  Nicholas  Torelli  requested  that  we  add  a  candidate  EM  on 
Personnel  Competency.  Some  prospective  candidates  were  identified;  USC  evaluated  each  and  mapped 
the  results  to  an  expanded  EM  performance  framework.  Given  that  MIT  was  unable  to  participate  in  the 
EM  task,  we  reallocated  its  evaluation  functions  in  the  revised  matrix,  Table  11,  and  added  funds  for  the 
remaining  performers. 


Candidates  Added 

•  Air  Force  Probability  of  Program  Success  (PoPS)  Framework. 
(https://acc.dau.mil/CommunityBrowser.aspx?id=24415) 

A  well-organized,  thorough  set  of  evaluation  criteria  in  wide  use  in  different  versions  by  the  Army, 
Navy,  and  Air  Force.  We  used  the  Air  Force  version  because  it  came  to  our  attention  first,  and  the 
others  were  similar. 

•  National  Research  Council  Pre -Milestone  A  and  Early-Phase  Systems  Engineering  study  Pre- 
Milestone  A/B  Checklist,  The  National  Academies  Press,  2008 

Twenty  questions  well-correlated  with  the  study’s  findings. 

•  Stevens  Leading  Indicators  of  Program  Success  and  Failure  Framework  (charts  by  Mark 
Weitekamp  and  Dinesh  Verma  posted  on  the  EM  task  collaboration  site) 

A  set  of  critical  success  factors  based  on  the  SADB  data  and  Program  Support  Team  Leader  (PSTL) 
interviews. 

•  Personnel  Competency  EM 

Primary  candidates  were  the  International  Council  on  Systems  Engineering  (INCOSE)  Systems 
Engineering  Certification  criteria  and  a  candidate  framework  identified  by  Dr.  Ken  Nidiffer  of  Carnegie- 
Mellon  University  /  Software  Engineering  Institute  (CMU/SEI).  At  the  INCOSE  Workshop  on 
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February  1,  the  INCOSE  leads  for  the  INCOSE  Systems  Engineering  Certification  criteria  agreed  to 
collaborate  in  providing  their  framework.  Dr.  Nidiffer  arranged  for  similar  access  to  the  other 
framework. 


Candidates  Dropped 

•  IBM-Stevens  Complexity  Point  paradigm  (Barker- Verma,  2003) 

More  focused  on  cost  estimation;  limited  number  of  EMs. 

•  USC-MIT  COSYSMO  Cost  Drivers  and  Risk  Analyzer  (Madachy  and  Valerdi,  2007) 
More  focused  on  cost  estimation;  EMs  mostly  covered  by  other  candidates. 

•  USC  Award  Fee  Criteria  for  Spiral  Developments  (Reifer-Boehm,  2006) 

More  focused  on  system  of  systems  acquisition;  EMs  mostly  covered  by  other  candidates. 

•  DAI  J-Fraunhofer  Center  Best  Practices  Repository  (Shull  and  Turner,  2005) 

EMs  largely  keywords;  most  covered  by  other  candidates. 

•  NUWC  Open  System  Engineering  Effectiveness  Measurement  (Kowalski  et  al.,  1998) 
Focused  mostly  on  an  aspect  of  net-centric  services;  somewhat  dated. 


Table  11.  Revised  EM  Evaluation  Assignments 


Candidate  EM 

USC 

Stevens 

FC-MD 

UAH 

PoPS  Leading  Indicators  (Lis) 

X 

X 

X 

INCOSE  Lis 

X 

X 

Stevens  Lis 

X 

X 

X 

SISAIG  Lis 

X 

X 

X 

NRC  List 

X 

X 

X 

SEI-CMMI 

X 

X 

X 

USC  AP-Feasibility 

X 

X 

X 

UAH  Team  Effectiveness 

X 

X 

X 

103 


Pers.  Competency 

X 

X 

3.  Project  Schedule 

On  January  9,  we  had  an  EM  task  planning  review  activity  in  which  we  rebaselined  the  proposed 
schedule  for  perfonning  and  coordinating  the  EM  task.  The  primary  changes  from  the  December  20 
Version  2  were  to  add  specific  survey/interview  and  coverage  matrix  deliverables;  to  change  the  January 
29-30  workshop  to  a  SERC-internal  teams-and-collaborators  workshop  on  interim  results;  to  have  a 
SERC-extemal  workshop  proposed  for  the  March  30-April  3  week  in  the  District  of  Columbia  (DC)  area 
on  the  EM  recommendations  for  pilot  evaluation;  to  have  a  piloting  readiness  review  in  the  DC  area 
May  6-7,  and  to  have  a  single  round  of  pilot  evaluations  between  May  1 1  and  July  10. 

At  the  January  29-30  workshop,  we  converged  on  March  31 -April  1  as  the  dates  for  the  next  workshop. 
We  adjusted  the  subtask  times  to  avoid  known  conflicts  with  holidays  or  major  conferences  and  to 
capitalize  on  neighboring  systems  engineering  community  events. 

With  respect  to  weekly  and  monthly  battle  rhythms,  we  converged  on  Fridays  1 1  AM  -  noon  Eastern 
Standard  Time  /  8-9  AM  Pacific  Standard  Time  for  both  weekly  tag-ups  and  monthly  sponsor  progress 
reports,  with  the  EM  task  doing  the  first  30  minutes  and  the  MPT  task  the  second  30  minutes.  At  the 
May  7  meeting,  the  sponsors  indicated  that  they  would  have  Chris  Miller  participate  in  the  weekly 
meetings  instead  of  having  monthly  sponsor  progress  reports. 

The  May  6  workshop  identified  several  improvements  that  would  be  needed  to  make  the  prototype 
Systems  Engineering  Perfonnance  Assessment  Tool  (SEPAT)  and  Systems  Engineering  Competency 
Assessment  Tool  (SECAT)  ready  for  piloting.  We  rescheduled  the  pilots  to  start  in  June,  and  moved 
back  the  analysis  of  the  pilots  and  pre-delivery  workshop  to  fit  within  the  extended  completion  date  of 
September  30.  The  September  workshop  date  was  approximate,  based  on  sponsor  availability.  It  was 
later  adjusted  to  September  8. 


Table  12.  EM  Task  Schedule  and  Results 


Period 

Activity 

Results 

12/8/08  - 
1/28/09 

Candidate  EM  assessments,  surveys, 
interviews,  coverage  matrix 

Initial  survey  results,  candidate  EM 
assessments,  coverage  matrix 

1/29/09  - 
30/09 

SERC-internal  joint  workshop  with 
Methods,  Processes,  and  Tools  (MPT) 

Progress  report  on  results,  gaps,  plans 
for  gap  follow-ups 

2/1/09  - 
3/27/09 

Sponsor  feedback  on  results  and  plans; 
sponsor  identification  of  candidate  pilot 
organizations;  execution  of  plans; 
suggested  EMs  for  weapons  platform 
(WP)  pilots 

Identification  of  WP  pilot  candidate 
organizations  at  Contractor,  PEO-PM. 
Oversight  levels;  updated  survey,  EM 
evaluation,  recommended  EM  results 
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3/31/09  — 
4/1/09 

SERC-external  joint  workshop  with 

MPT,  sponsors,  collaborators,  pilot 
candidates,  potential  EM  users 

Guidance  for  refining  recommended; 
EMs;  candidates  for  pilot  EM; 
evaluations 

4/7/09  - 
5/1/09 

Tailor  lead  EM  candidates  for  WP  pilots; 
SADB-based;  evaluations  of  candidate 

EMs 

Refined,  tailored  EM  candidates  for 

WP  pilots  at  contractor,  PEO-PM, 
oversight  levels;  pilot  evaluation 
guidelines 

5/6/09 

Workshop  with  sponsors,  collaborators, 
pilot  candidates,  stakeholders;  select  WP 
EM  pilots 

Selected  pilots;  guidance  for  final 
preparation  of  EM  candidates  and 
evaluation  guidelines 

5/11/09  - 
6/12/09 

Revise  EM  tools  based  on  feed  back; 
prepare  pilot  users’  guide;  Line  up  pilot 
projects 

Most  pilots  ready  to  begin 
experimental  use;  some  completing 
preparations 

6/15/09  — 
8/14/09 

EM  pilot  experiments;  analysis;  and 
evaluation  of  guidelines  and  results; 
refinement  of  initial  SADB  evaluation; 
results  based  on  EM  improvements 

EM  pilot  experience  database  and 
survey  results;  refined  SADB  EM 
evaluations 

8/17/09  — 
9/14/09 

Analyze  EM  pilot  and  SADB  results; 
prepare  draft  report  on  results, 
conclusions,  and  recommendations 

Draft  report  on  general  EM  evaluation 
results,  conclusions,  and 
recommendations  for  usage  and 
research/transition/education 
initiatives 

9/15/09 

Workshop  on  draft  report  with  sponsors, 
collaborators,  EM  evaluators, 
stakeholders 

Feedback  on  draft  report  results, 
conclusions,  and  recommendations 

9/16/09  — 
9/30/09 

Prepare,  present,  and  deliver  final  report 

Final  report  on  MDAP  EM  evaluation 
results,  conclusions,  and 
recommendations  for  usage  and 
research/transition/  education 
initiatives 
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